{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 金融风控-贷款违约预测 建模调参\n",
    "\n",
    "### 学习任务\n",
    "\n",
    "学习在金融分控领域常用的机器学习模型\n",
    "\n",
    "学习机器学习模型的建模过程与调参流程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 理论基础\n",
    "\n",
    "### 1.逻辑回归\n",
    "\n",
    "sigmoid函数\n",
    "\n",
    "$$ logi(x) = {1 \\over {1+e^{-x}}} $$\n",
    "\n",
    "sigmoid是一种分类方法，主要用于两分类问题（即输出只有两种，分别代表两个类别）\n",
    "\n",
    "sigmoid函数的特点：[-5,5]之间快速变化由-1到1,；奇函数；当x≥0 时,y≥0.5,分类为1，当 x<0时,y<0.5,分类为0\n",
    "\n",
    "代价函数：适合于逻辑回归的代价函数是对hx取负对数\n",
    "\n",
    "![avatar](https://img-blog.csdn.net/20151014130511356?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQv/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center)\n",
    "\n",
    "梯度下降法：调整参数θ使得代价函数J(θ)取得最小值，形像的说也就是从山顶一步一步的向下挪动到极小值处。这其中挪动的步长被称为学习率；越接近极值点处越小的步长，也就是使用可变学习率的原因。\n",
    "\n",
    "### 2.树模型\n",
    "\n",
    "|算法|思路|\n",
    "|--|--|\n",
    "|ID3算法|从根结点(root node)开始，对结点计算所有可能的特征的信息增益，选择信息增益最大的特征作为结点的特征，由该特征的不同取值建立子节点；再对子结点递归地调用以上方法，构建决策树；直到所有特征的信息增益均很小或没有特征可以选择为止，最后得到一个决策树\n",
    "|C4.5|与ID3类似，只不过信息增益变成信息增益率|\n",
    "|C5.0|由C4.5改进，适用于处理大数据集，采用Boosting方式提高模型准确率|\n",
    "\n",
    "### 3.集成模型\n",
    "\n",
    "|Bagging|Boosting|\n",
    "|--|--|\n",
    "|随机森林|Adaboost、GBDT、XgBoost、LightGBM|\n",
    "|训练集是从原始集中有放回的选取，所以从原始集中选出的各轮训练集之间是独立的|每一轮的训练集不变，只是训练集中每个样本在分类器中的权重发生变化。而权值是根据上一轮的分类结果进行调整\n",
    "|均匀取样，所以每个样本的权重相等|根据错误率不断调整样本的权值，错误率越大则权重越大|\n",
    "|所有预测函数的权重相等|每个弱分类器都有相应的权重，对于分类误差小的分类器会有更大的权重|\n",
    "|各个预测函数可以并行生成|各个预测函数只能顺序生成，因为后一个模型参数需要前一轮模型的结果|\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 贪心调参方法；\n",
    "\n",
    "先使用当前对模型影响最大的参数进行调优，达到当前参数下的模型最优化，再使用对模型影响次之的参数进行调优，如此下去，直到所有的参数调整完毕。\n",
    "\n",
    "缺点：可能会调到局部最优而不是全局最优\n",
    "\n",
    "\n",
    "### 网格调参方法；\n",
    "\n",
    "sklearn 提供GridSearchCV用于进行网格搜索，只需要把模型的参数输进去，就能给出最优化的结果和参数。\n",
    "\n",
    "### 贝叶斯调参方法\n",
    "\n",
    "贝叶斯调参的主要思想是：给定优化的目标函数(广义的函数，只需指定输入和输出即可，无需知道内部结构以及数学性质)，通过不断地添加样本点来更新目标函数的后验分布(高斯过程,直到后验分布基本贴合于真实分布）。简单的说，就是考虑了上一次参数的信息，从而更好的调整当前的参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\r\n",
    "import numpy as np\r\n",
    "import warnings\r\n",
    "import os\r\n",
    "import seaborn as sns\r\n",
    "import matplotlib.pyplot as plt\r\n",
    "\"\"\"\r\n",
    "sns 相关设置\r\n",
    "@return:\r\n",
    "\"\"\"\r\n",
    "# 声明使用 Seaborn 样式\r\n",
    "sns.set()\r\n",
    "# 有五种seaborn的绘图风格，它们分别是：darkgrid, whitegrid, dark, white, ticks。默认的主题是darkgrid。\r\n",
    "sns.set_style(\"whitegrid\")\r\n",
    "# 有四个预置的环境，按大小从小到大排列分别为：paper, notebook, talk, poster。其中，notebook是默认的。\r\n",
    "sns.set_context('talk')\r\n",
    "# 中文字体设置-黑体\r\n",
    "plt.rcParams['font.sans-serif'] = ['SimHei']\r\n",
    "# 解决保存图像是负号'-'显示为方块的问题\r\n",
    "plt.rcParams['axes.unicode_minus'] = False\r\n",
    "# 解决Seaborn中文显示问题并调整字体大小\r\n",
    "sns.set(font='SimHei')\r\n",
    "\r\n",
    "\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一个函数减小读取数据占用内存的大小，核心是用iinfo输出对应数据类型的最大值最小值，然后将落在其中的适合值转化为该类型数据。\n",
    "\n",
    "总而言之是不影响结果的前提下降低数据的精度\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def reduce_mem_usage(df):\r\n",
    "    start_mem = df.memory_usage().sum() \r\n",
    "    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\r\n",
    "    \r\n",
    "    for col in df.columns:\r\n",
    "        col_type = df[col].dtype\r\n",
    "        \r\n",
    "        if col_type != object:\r\n",
    "            c_min = df[col].min()\r\n",
    "            c_max = df[col].max()\r\n",
    "            if str(col_type)[:3] == 'int':\r\n",
    "                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\r\n",
    "                    df[col] = df[col].astype(np.int8)\r\n",
    "                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\r\n",
    "                    df[col] = df[col].astype(np.int16)\r\n",
    "                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\r\n",
    "                    df[col] = df[col].astype(np.int32)\r\n",
    "                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\r\n",
    "                    df[col] = df[col].astype(np.int64)  \r\n",
    "            else:\r\n",
    "                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\r\n",
    "                    df[col] = df[col].astype(np.float16)\r\n",
    "                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\r\n",
    "                    df[col] = df[col].astype(np.float32)\r\n",
    "                else:\r\n",
    "                    df[col] = df[col].astype(np.float64)\r\n",
    "        else:\r\n",
    "            df[col] = df[col].astype('category')\r\n",
    "\r\n",
    "    end_mem = df.memory_usage().sum() \r\n",
    "    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\r\n",
    "    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\r\n",
    "    \r\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Memory usage of dataframe is 300800080.00 MB\n",
      "Memory usage after optimization is: 72834896.00 MB\n",
      "Decreased by 75.8%\n"
     ]
    }
   ],
   "source": [
    "data = pd.read_csv('data/data54058/train.csv')\r\n",
    "data = reduce_mem_usage(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于数据问题，这里去掉了所有和sample相关的行列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "#建模之前的预操作\r\n",
    "from sklearn.model_selection import KFold\r\n",
    "# 分离数据集，方便进行交叉验证\r\n",
    "\"\"\"\r\n",
    "X_train = data.loc[data['sample']=='train', :].drop(['id','issueDate','isDefault', 'sample'], axis=1)\r\n",
    "X_test = data.loc[data['sample']=='test', :].drop(['id','issueDate','isDefault', 'sample'], axis=1)\r\n",
    "y_train = data.loc[data['sample']=='train', 'isDefault']\r\n",
    "\"\"\"\r\n",
    "X_train = data.drop(['id','issueDate','isDefault'], axis=1)\r\n",
    "X_test = data.drop(['id','issueDate','isDefault'], axis=1)\r\n",
    "y_train = data[ 'isDefault']\r\n",
    "\r\n",
    "# 5折交叉验证\r\n",
    "folds = 5\r\n",
    "seed = 2020\r\n",
    "kf = KFold(n_splits=folds, shuffle=True, ran\n",
    "dom_state=seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集的划分方式：\n",
    "\n",
    "#### 留出法\n",
    "\n",
    "留出法是直接将数据集D划分为两个互斥的集合，其中一个集合作为训练集S，另一个作为测试集T。为了保证数据分布的一致性，通常我们采用分层采样的方式来对数据进行采样。\n",
    "\n",
    "#### k折交叉验证\n",
    "\n",
    "通常将数据集D分为k份，其中k-1份作为训练集，剩余的一份作为测试集，这样就可以获得k组训练/测试集，可以进行k次训练与测试，最终返回的是k个测试结果的均值。交叉验证中数据集的划分依然是依据分层采样的方式来进行。\n",
    "\n",
    "#### 自助法\n",
    "\n",
    "我们每次从数据集D中取一个样本作为训练集中的元素，然后把该样本放回，重复该行为m次，这样我们就可以得到大小为m的训练集，在这里面有的样本重复出现，有的样本则没有出现过，我们把那些没有出现过的样本作为测试集。\n",
    "\n",
    "进行这样采样的原因是因为在D中约有36.8%的数据没有在训练集中出现过。留出法与交叉验证法都是使用分层采样的方式进行数据采样与划分，而自助法则是使用有放回重复采样的方式进行数据采样\n",
    "\n",
    "### train_test_split api\n",
    "\n",
    "|方法|用途|\n",
    "|--|--|\n",
    "|train_target|所要划分的训练样本|\n",
    "|test_size|样本占比|\n",
    "|random_state|是随机数的种子|\n",
    "|stratify|按照谁的样本分布，可以为none或者一个数组，none的情况就同训练集，数组则是指定比例|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 200 rounds\n",
      "Early stopping, best iteration is:\n",
      "[127]\tvalid_0's auc: 0.719521\n"
     ]
    }
   ],
   "source": [
    "\"\"\"对训练集数据进行划分，分成训练集和验证集，并进行相应的操作\"\"\"\r\n",
    "from sklearn.model_selection import train_test_split\r\n",
    "import lightgbm as lgb\r\n",
    "# 数据集划分\r\n",
    "X_train_split, X_val, y_train_split, y_val = train_test_split(X_train, y_train, test_size=0.2)\r\n",
    "train_matrix = lgb.Dataset(X_train_split, label=y_train_split)\r\n",
    "valid_matrix = lgb.Dataset(X_val, label=y_val)\r\n",
    "\r\n",
    "params = {\r\n",
    "            'boosting_type': 'gbdt',\r\n",
    "            'objective': 'binary',\r\n",
    "            'learning_rate': 0.1,\r\n",
    "            'metric': 'auc',\r\n",
    "            'min_child_weight': 1e-3,\r\n",
    "            'num_leaves': 31,\r\n",
    "            'max_depth': -1,\r\n",
    "            'reg_lambda': 0,\r\n",
    "            'reg_alpha': 0,\r\n",
    "            'feature_fraction': 1,\r\n",
    "            'bagging_fraction': 1,\r\n",
    "            'bagging_freq': 0,\r\n",
    "            'seed': 2020,\r\n",
    "            'nthread': 8,\r\n",
    "            'silent': True,\r\n",
    "            'verbose': -1,\r\n",
    "}\r\n",
    "\r\n",
    "\"\"\"使用训练集数据进行模型训练\"\"\"\r\n",
    "model = lgb.train(params, train_set=train_matrix, valid_sets=valid_matrix, num_boost_round=20000, verbose_eval=1000, early_stopping_rounds=200)\r\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "未调参前lightgbm单模型在验证集上的AUC：0.7179368297692927\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAH1CAYAAABvDS3zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4VGXC/vHvTHrvjYReDiXSEcQKiogVRUBEfC3r2juWd99dd3/7vrs27KKrLvZesYAiIIoVVED6oZf03uvMnN8fiS4qkAGSnEnm/lwXF5PDlDtDMnPPc57zHIdlWYiIiIj/cdodQEREROyhEiAiIuKnVAJERET8lEqAiIiIn1IJEBER8VMqASIiIn5KJUBERMRPBdodQER8j2EYu4AUwA1UAZ8A15mmWdX872OB/wNGAR5gOXCHaZob97mPaODvwHlAPJAPfAj8n2maRe31vYjIgWkkQEQO5CzTNCOBocAw4L8BDMM4BvgUeB/oAvQEfgK+NgyjV/N1goGlwCDgNCAaOAYoBo5u329DRA5EIwEiclCmaeYZhrGIpjIAcB/wommaj+xztT8bhjEC+BtwcfOfbsC4n0cPgALgf9sntYh4QyMBInJQhmFkAJOAbYZhhANjgbf2c9U3gQnNl08BPtmnAIiID1IJEJEDmW8YRiWwl6ZP8X+lad++E8jdz/VzgcTmywkHuI6I+BCVABE5kMmmaUYBJwH9aXqDL6VpImDafq6fBvw84a/4ANcRER+iEiAiB2Wa5hfA88Ac0zSrgW+Bqfu56jSaJgMCLAEmGoYR0S4hReSwaGKgiHjjYWCXYRhDgDuBRYZhbAaeo+l15FaaZv+Par7+S8CVwDuGYdwEbAHimretMU1zYTvnF5H90EiAiLTINM1C4EXgLtM0vwIm0nT8fy6wm6ZDCI8zTXNr8/XraZocuBlYDFQAK2napbCi3b8BEdkvh2VZdmcQERERG2gkQERExE+1y5wAwzDmAFOAHsBRpmmu3891AoBHaVpdzALuMU3z3+2RT0RExB+110jAfOAEmvYdHshMoA/Ql6YJRn8zDKNH20cTERHxT+0yEtA8kQjDMA52tenAM6ZpeoBCwzDm03QY0v1ePkwITTOTc2k66YmIiEhnFkDTehzfA/WHcwe+dIhgN349UrAH6HoItx8FfNmqiURERHzf8cBXh3NDXyoBRyoXoLS0Go9HRzy0lYSESIqLtRx8W9Pz3Pb0HLc9PcfgsSwKSmrJL60ht7iGovI6tuwtpa7h4APWEWFBRIUF0ej20CUhgsjwIIIDAwCICw8g/quPiKopZfA9/4AjWKLbl0rAHqA7TcMa8PuRgZa4ATweSyWgjen5bR96ntuenuO25y/PcX2Dm5ziavJLa9iVW8nO3AqKyuuorm2kweUBwOlwkBgTSkJMKNERwfRKiyYpNoygQCfxUaEEBzlJiAklPCQQh8NxwMdy19aS/UE+USef8sumw83tSyXgLeAKwzDepenkI5NpGuIQERHxGbX1LnblVbK3oIo1WwspqainsKyWn+uOA4iNCiE0OIC+GTEM6hFPclwYvbpEE9T8af5weOqbdvsHhIXR9Y4/ERAcdMTfS3sdIvgoTauLpQJLDMMoNk1zkGEYC2lagewHmpYZHQ1sbb7Z303T3Nke+URERH6rvtHN7rymT/W5xTXszqsku6gKl/s/oxtxUSHER4UwNjOVLokRpCaEkxwbRnDQ4b/Z74+nro7sxx7GERRE+o234Ahsnbfv9jo64Abghv1sP32fy27g6vbIIyIi8ltFZbWYe8vYnl3OzrxKdudV/urf0xMjGNQjnpT4cAb2iKdrciRxUSFtnstdW0v2Iw9St2M7qZdfcdBdBYfKl3YHtBm320VpaSEuV4PdUTq8ggInHo/H7hitLjAwmLi4JAIC/OJXQkSA3XmVbNxdwo6cCnbkVFBa2TTcHh4SSFpiOJPGdCMjKZIeqVGkxIfjbMU3X2+5a6rJfvgB6nbvJu2PVxM1clTLNzoEfvGKV1paSGhoOBERqa3aoPxRYKATl6tzlQDLsqiurqC0tJDExDS744hIG3F7PGzLKmfdjhK+WptDRU0jAAnRofROj6Fvegx9MmLonhplyxv+/uQ+/RR1u3fT5apriRw2vNXv3y9KgMvVoAIgB+RwOIiIiKaqqszuKCLSiizLYk9+FRt2lbAzt4JNu0qpqXcB0LtLNKeN7s7ogSntMqR/uJLOn4qr7BQiMge3yf37RQkAVADkoPTzIdI5eDwWO3Iq+GpdLut2FP8yxJ8YE8qQPgkM7p3IUb0SCA/13bc/V3kZld+vJPbkCYRkdCUk41DWzTs0vvssiIiItMCyLLIKq/l+cz7ZhdVszSqnqrYRB5ASH87Ucb0Z1T+ZxJgwu6N6pbG0lKw59+IqKyVyyDCCkpLa9PFUAtrZrbfewPHHn8Dkyef/ss2yLKZNm8yf/nQXw4aNOOBtr7vuj8yYMYtjj93/8gm7du3kooumcv31NzN9+sxftv/jH3+jf/8BTJky/Zdtjz/+MGFhYVx++ZUAbNq0gaeemktOTjbR0TGEh4dz2WV/ZOjQI98H5Xa7efjhOaxY8Q0Oh4OLLrqEs86a/LvrLVu2hBdeePaXrwsL8xkyZDj//Of9NDQ0cOedt2KaGwFYsGCpV7cDeOml5/j0049xu90MHJjJ7bf/D8HBwUf8fYmIPUor61m1pZAdORWs3V5EdV3TEH9woJPhRhJDeicyoEcc0eEd6/e8sbiIrDn34q6sJOOm2W1eAEAloN2dccbZvP76y78qAatX/4jT6TjiN9wFCz5gxIhRLFz44a9KQEu2b9/GbbfdxF/+8ndGjz4GgOzsLLZuNY8oz88+/fRjsrP38vrr71FeXs5ll81k5MijSUvr8qvrjRt3CuPG/bICFpdeeiETJkwEwOl0MmPGRcTGxnLTTdd4fbuVK79jyZJFPP30C4SGhnLfff/gjTdeZdasS1rlexOR9tHo8rB+ZzFfr8tjzdYiPJZFVHgQPdOiMbrFMnpgSof5tL8/DYUFZM25F09NDem33EZYr97t8rh+VwK+XpfLV2sPe5nlgzpucBrHHnXw2eXHH38iDzxwN7t27aRHj55A05v36aefhcPh4IcfVvLMM0/S0FCP2+3m4osv45RTJrb42C6Xi08/Xcjcuf9m9uwb2LRpAwMGDPIq9yuvvMCZZ57zSwEASE/PID09w6vbt+SzzxZz1lmTcTqdxMXFcfzxJ7Js2RIuvPDiA97GNDdTWFjAccedCEBgYCCjRo0mNzfnoI/129tt27aFwYOHERbW9OIwZsxY5s17SiVApAMoqahjS1YZG3eW8uOWQmrrXUSEBnLKyAyOH5xGelKk3RFbTf3evVgNDWTMvoPQ7j3a7XH9rgTYLSgoiAkTJrFw4Qdcc82N1NRU8+WXX/Dyy28C0K9ff5544t8EBARQUlLM5ZfP4uijjyE6Ovqg9/vtt1+RkdGNjIyuTJp0JgsWfOB1CdiyZTMnnXSyV9fduXMHf/nLn/b7b6NGjebaa2/83fb8/DxSU/9TjlJSUikoyD/o4yxY8D4TJkwiKOjQlsX87e0MYwAffDCfsrIyIiMj+eyzxeTl5R3SfYpI+ymrqmfjrhK+WpvL5j1NR+wEBzkZ3i+Jo/unkNkrnsAAp80pW4+nsQFnUDBRw0cQMXAgztD2Hc3wuxJw7FEtf1pva2eccTazZ1/PlVdex9KliznqqCEkJ6cAUFZWyt13/52srD0EBARSUVHOnj27ycw86qD3uWDBB0yadCYAp512BpdeOpPrr7+FkJCQA858P5wZ8T179uL551895NsdioaGBhYvXsRjjz11xLcbMWIU5503lVtuuZbg4BBGjBhFQMCK1o4sIkfA47FYv7OY5T/lsmpLIdA0m//sY3swsEc8PdOiCQrsPG/8P6vP2kv2Iw+RcvElRBw1uN0LAPhhCfAFffv2IyEhie+++4aFCz9g6tQLf/m3Bx64h2OPPYF//vN+HA4HF1xwHg0N9Qe9v5KSYlau/I4tW0yef/7fANTV1fHFF59x6qmTiI2Npby8/Fe3KS8v+2W4v1+//mzatIETTjipxeyHMxKQkpJKXl7uLyMTvx0Z+K3ly5fRpUs6ffr0bTGPN7ebNm0G06bNAGDp0sW/7IYREft4LIuNO0v4cUshKzcVUFvvIjwkkEljupHZIx6jWxxOZ+c9dLduz26yHrwfR2Bgu0wAPBCVAJucccbZPPvs0+Tn53L88Sf+sr2yspK0tDQcDgfff/8d2dl7W7yvTz5ZwEknncxdd/3vL9sWL/6Ejz56n1NPncSoUaN55JEHOf/8C4iOjiY/P4+VK7/9Zb/4jBmzuPnmaxg+fCSjRo0GICcne7+7CQ5nJGDcuFP48MP5nHjieMrLy/nyyy+YO/eZA15/wYIPOOOMsw/pMQ52u+LiIhISEqmoqOCVV57nkkuuOOT7FpEjY1kWucU1mHvL2FtQxfodxRSV1xHgdDC4dwJjM1MZ0iexUw31H0jdzh1kPTQHZ2gYGbPvIDg52bYsKgE2mTDhNObOfYSzzz73V/u9r776Oh544F7mzXuaAQMG0rt3y5+GFy78kGuvvelX244//iTmzLmb3NwcRo0aw+TJ53H99VficDhwOp3cdNNtdOvWA2gambjnnod45pknuP/+fxIaGkpsbByXX35Vq3yvEyeezsaN67nggnMBuOSSP9ClSzoA8+e/TVFREX/4Q9Nj5efnsW7dT/z97/f87n7+8IeLKSzMp7KyknPPPZ3Ro4/hzjv/0uLtbr75WjweC5fLxZQp07wa8RCR1rEzt4L1O4pZs62Inbn/OSFP7y7RnDm2B6MHpBAS3Lpn3PNljYWFZD14PwERkWTMvp2gRPtGAQAclmW1fK2OoQews7i4Co/n199TXt5uUlO72xKqs+mM5w74mS/9nCQlRVFYWNnyFeWw6TluOxXVDazYmM+KTQXsyGnaFZkUG8q4YRkM6hlPWkK4X3zi3x/LsihZ8CHRY48lKD7hiO7L6XSQkBAJ0BPYdTj3oZEAERE5YhU1DazcmM+m3aWs3loEQNeUKKaN68MII4mk2I57DH9rqNm8icC4OIJTUkk489B3d7YVlQARETksLreHFRvz+XpdLluzynE3j8KeMiKD4wanMSKzi0ZbgOr1a8mZ+xhhxgAybrrF7ji/ohIgIiJec7k9fLchn23Z5Xy/OZ/aejfJcWGcPCKDYwal0i0lUifk2kfVT2vIffJxgtO6kHa5701K9psSYFmWfjDlgDrR3BiRVldd18iGnSUsW5XNzrwKGhqb5gUdPSCZMYNSGdwroVMfzne4Klf9SO5TTxDStRsZN88mICLC7ki/4xclIDAwmOrqCiIiolUE5Hcsy6K6uoLAwI51shGRtlRWVc+PZiHZRdV8vS6XRpeHiNBAjuqVwLGZaZ1u5b7WZlkWZUsXE9qjJ+k33kJAeLjdkfbLL0pAXFwSpaWFVFWV2R2lw3M6nXg8ne/ogMDAYOLi7D1UR8QXFJTW8Pbn2/nBLPxl25DeCUwa053e6dEEOPXG3xLL48HhdNLluhtxOLBlJUBv+UUJCAgIJDHR3qWCOwsdViXS+Xgsi827S1m2OpsfzUICnA5OGZnB4N4J9O8Wp0/8h6D8q+VUfPsN6TfcTECY7775/8wvSoCIiPxefkkNq7cW8e2GPPYWVBEeEsipo7oyfng6yXG+OXzty8o+/4yCl18kfFAmdJBdzyoBIiJ+pK7Bxdfr8vhybQ578qsA6JIYwSWT+vvd6n2tqXTJYgpff4WIwUNIu/panEEdY46RSoCIiB8oKqvli59y+GxVNrX1LtISwpk6rjfD+yWRok/9R6Rs2WcUvv4KkcNHkPbHq3EEdpy31o6TVEREDklBWS2rzEJWbSlkW3bT8r09UqOYclJvBnSPw9lBhqx9XXj//sSMG0/y9As7VAEAlQARkU7FsizMPWUsW53N95sLAEiJC+Oc43py9IBk0hJ871j1jsiyLGrWryM88yiC07qQMvNiuyMdFpUAEZFOoL7RzSqzkCU/7mVnbiVhIQGMH57OuOEZpCfqjb81WZZF0TtvUfrJQtKuupaokaPsjnTYVAJERDqw8qp6lq/N5dOVe6iuc5EQHcJFp/bj2Mw0TfJrA5ZlUfjGa5Qt+ZSYk8YTOXyE3ZGOiEqAiEgHtC27nMXf72XNtiIaXR4GdI9j0uhuDOgRpwV92ojl8VDw2suUL/uM2FMmkDT9wg6/Cq1KgIhIB1Fe3cBXa3NYs7WI7TkVhAYHMHpACpPGdNO+/nZQv3cP5cu/IO6000mcMrXDFwBQCRAR8WlVtY18smIPW7LK2J5djmVBt5RIppzYi3HD0gkPDbI7ot8I7d6D7nf9P4K7pHeKAgAqASIiPqmypoGPv9vD0lVZNLo8JESHMGFkV44bnEZGUqTd8fyG5XaT9/w8IocOI2rEKELSM+yO1KpUAkREfIRlWWzaXconK/awfmcJ0HS63tPHdKdbSpTN6fyP5XKR+8y/qPrxB0Iyutodp02oBIiI2MxjWazYkM+yNdlsyyonNDiACSO7cuxRqXrzt4mnsZHcf82l+qc1JE2fQdyEiXZHahMqASIiNvFYFmu2FjH/y51kFVYR4HQw4+S+HD8kjdBgvTzbxXK5yJn7KDXr15E882Jix423O1Kb0U+ZiEg783gsvlqXy8LvdlNQWktMZDCXTurPMZmpOm2vLwgIICQ9g6gRI4k5/kS707QplQARkXaUXVjFvAWb2JVXSXpiBJefMYCjB6QQFKg3f7t56upwlZURnJpK0tTpdsdpFyoBIiLtIKugio9X7Ob7zYWEBDm5dFJ/jhuc1mkONevo3DU1ZD/yIK6SYnr8416cwR3jVMBHSiVARKQNZRdV88FXO/l+cwFBgU7GZqZy1tgeJMSE2h1Nmrmrq8l6aA71e/eQ9ser/KYAgEqAiEib2LCzhNeWbiW3qJrg4ABOHdWVSaO7ERMZYnc02Ye7spKsh+bQkJNNl6uvI3LoMLsjtSuVABGRVuLxWPy0vYhFK/awJauc2Mhgjh+Sxnkn9CY6wn8+XXYkRR+811QArruBiMzBdsdpdyoBIiKtYGduBc8u3ER2YTXREcFMObEXp47qSlCgzuTny5LOn070mLGE9e5jdxRbqASIiBwmy7LYsKuEr9bm8v3mAsJDApk+vg/jh6frzd+HNZaWUvTOm6RcdDHO0DC/LQCgEiAicli+XZ/HO8u3U1JRT2hwABNHdePMsd11Qh8f11hcRNace3FXVdFw6mmEdutudyRbqQSIiByCkoo63l2+g2/W55EUG8pFp/bj2Mw0QoL1yd/XNRQWkHX/vXjqakm/+Ta/LwCgEiAi4pWi8lo+XbmXz1ZlY1kWp47qypQTe2nYv4NoyM8ja869eBoayLj1dkK797A7kk9QCRAROYjaehdLftjLB1/vwrLgmEEpnHlsD1Liwu2OJofC4SQgKpr0S/9ASNfOeUbAw6ESICKyHzV1LhZ+t5vPV2dTU+9iaJ9ELji5D8l68+9QGouLCYyPJzg5mW5/+ZtWaPwNlQARkX3U1rv4am0u7325g7oGN4N6xnPmMd0xusXZHU0OUd3uXWQ9eD+x408h8ZxzVQD2QyVARARwuT0sW53Nh1/voqq2kT7pMZx/Um/6dY21O5ochtodO8h+eA7O0DCijznW7jg+SyVARPzepl1NS/xmFVbTMy2Ka8/N1Cf/Dqx221ayH36AgKgoMmbfQVBCot2RfJZKgIj4rb0FVbz9+XbW7SgmOiKYK88exOiBKXbHkiPgqasl+7GHCYiJIePWOwiKj7c7kk9TCRARv1Nb7+KljzfxzmdbCQp0Mvm4npw2uhvBQTrcr6NzhoaR9serCUnPIDBWu3JaohIgIn7DY1l8tyGPt5Ztp7y6gVH9k7lwQj9idHKfDq963Vrc1VVEjxlLxKBMu+N0GCoBIuIXtuwt49UlW9iTX0VaQjh/uvRokiL15t8ZVK1ZTe6/5hKc0ZWoo8fgcDrtjtRhqASISKfW0Ojm3eU7+PT7vcREBHPxaQYnDO5CSko0hYWVdseTI1T54/fkPv0vQrt1J/2mW1UADpFKgIh0Wlv2lvHcx5vJL6nhmEGpzJrYj9Bgvex1FhUrviNv3tOE9uxF+k23EhAWZnekDke/DSLS6eQWV/Pu8h38aBYSFxXCTVMHM7i3DhPrbBrycgnr05f0G27GGRpqd5wOSSVARDqNmjoXry3dwrfr8wkMdHDGMd05fUx3wkL0UteZuGuqCQiPIOHsyeB24wjU/+/h0jMnIh1eo8vDT9uKeO/LHeSX1HLSsC6cNbYHMZEhdkeTVla2bCnFH8yn653/Q3BKKqgAHBE9eyLSoe3Oq+TpDzeQW1xDfHTT0H9mrwS7Y0kbKF28iMI3XiNi6DAC4/V/3BpUAkSkQ7Isi0Ur9/L259uJDAvk6smZDOubSGCAZod3RiUfL6DonbeIHDGStCuu0i6AVqJnUUQ6nMKyWl78ZDMbdpUyoHscfzhzIHFRGvrvrCpWfEfRO28RdfQYUi+/AkeAVnZsLSoBItJhuNwePv5uNwu+2w0WXHByXyaMzNApYju5yGHDSZw6nbgJE7UOQCtTCRCRDmHT7lJeXGSSX1LD4N4JzJzQj6RYHRfeWVmWRdnSJUSPHUtAeATxEyfZHalTUgkQEZ9WW+9i3oJNrNpSSERoINdMzmSEkaRP/52YZVkUvvEqZUsWg8dN3Kmn2R2p01IJEBGfZe4p5dmFmygqr2PCyK6cc1xPwkP1stWZWR4PBa+8RPkXy4g95VRiJ0y0O1Knpt8mEfE55dUNvLd8O1+tzSMyPIhbpg1lUE+dF76zszwe8l98joqvviTutNNJnDJVIz5tTCVARHzKlr1lPPHeOiprGzl+cBemjetNeGiQ3bGkHbirqqjZvIn4s84h4ezJKgDtQCVARHxCfYObBd/tZuG3u4mPDuF/zh9Cry7RdseSdmC53eBwEBgdTfe7/h8B4RF2R/IbKgEiYivLsvhk5R4WfLObmnoXI/snc/FEg8gwffr3B5bLRe5TT+KMiCDlvy5VAWhnKgEiYptGl5vXlm7j89XZDOwRx1lje2B0i7M7lrQTT2MDuU/OpXrtTyRdcKGG/22gEiAitli1pZCXPzUpq2rg1FFdmT6+j94E/IinoYGcuY9Ss2E9yRddTOxJ4+2O5JdUAkSkXdU1uHh1yVa+WptLWkI4fzhzIAN7aOa/v8n911xqNm4g5ZLLiDnuBLvj+C2VABFpN9uyy3ly/npKK+s5ZWQGU07sTUiQ1oH3R7EnTyBq1GiijxlrdxS/phIgIm3O5fbwyYo9vPflDuKjQrhz5nD6dY21O5a0M3dNDbXmJiKHjSBiUKbdcQSVABFpY6u2FPLaki0UV9QzuHcCfzxroI7790Pu6mqyHppDQ9ZeevzzPoLitQvIF6gEiEibqK138eaybXyxJofU+HCuPTeTYf2ScGryn99xV1aS9eD9NOTmkHb1dSoAPqTdSoBhGP2AF4AEoBi42DTNrb+5TjLwHNAVCAKWATeYpulqr5wicmTcHg/frM9j/pc7Ka9q4KRh6Vwwvg/B2vfvl1zl5WQ9eD+NBfl0ue5GIjKPsjuS7KM9T8z8L2CuaZr9gLnAU/u5zp+ATaZpDgYGAyOA89ovoogcib0FVfz5mRU8t3AzocEB3DFzGBdPNFQA/FjVmtU0FhaQfsPNKgA+qF1GApo/4Q8HJjRveg143DCMJNM0C/e5qgVEGYbhBEKAYCC7PTKKyJH5dkMeLy0yCQp0ctU5gxjZP1lD/37MsiwAYk88iYjMTIISEm1OJPvTXrsDugLZpmm6AUzTdBuGkdO8fd8S8L/AO0AuEAE8bprm14fyQAkJka2TWA4oKSnK7gh+oaM8zx6PxcufbOKtpVsZ0COe2TNHkBwfbncsr3SU57ijqcsvYPM99xF+3TUk9e4Fep59lq9NDJwKrAVOBqKAjw3DON80zbe9vYPi4io8Hqut8vm9pKQoCgsr7Y7R6XWU59myLF7+dAvLVmdz3FFpzJpo4HC7O0T2jvIcdzQNBQVkzbkHT10dlsej57gNOZ2OI/7g215zAvYC6YZhBAA0/92lefu+rgdeMU3TY5pmOfA+MK6dMorIIXC5Pby0yGTZ6mxOGtqFS0/vT1Bge04zEl/TkJfL3vv+iaehgYzZdxDVt4/dkaQF7fIba5pmAbAGmNG8aQaw+jfzAQB2AqcBGIYRDJwCrG+PjCLivcKyWu5+eRWfr8lh3LD0phEA7f/3aw2FBey9725we+h6252EdutudyTxQnvuDrgKeMEwjLuAUuBiAMMwFgJ3mab5A3AT8C/DMNYBATQdIvhMO2YUkRZ8v7mA5xZuwuFw8MezBjJmUKrdkcQHBMbGETF4CPETJxGc1sXuOOIlx88zODuBHsBOzQloW9qP2j588XluaHTz6pKtLP8ph24pkVw9OZOUuI4xAXB/fPE57ojq9uwmKC6egKjfT/7Tc9y29pkT0BPYdTj34WsTA0XEB+3Jr+TJ+evJL63lhCFdmDmhL0GBOvbf39Vu30b2ww8QZvQn/bob7Y4jh0ElQEQO6ruNebz4iYkFXDM5k5H9k+2OJD6gdusWsh95kICoaJJnXGR3HDlMKgEisl9ZhVW8vMhkS1Y56UkR3Hj+YBJjwuyOJT6gZvMmsh97mMC4ODJuvYOguDi7I8lhUgkQkd/5dn0ez328mZAgJ9PH9+HkERkEBujwPwHL46Hg9VcJSkgg49bbCYzRKaE7MpUAEfmFZVm89+VOPvpmF726RHPl2YNIitWnf/kPh9NJ+g034QgKIjAq2u44coRUAkQEgLKqeuZ9tJENu0oZ2T+Zy08fQEiwJv9Jk6rVq6j6aTUpF19KUHyC3XGklagEiPg5y7L4bmM+ry7eQk29i0ljunH+ib21+I/8ovKHleQ+8xSh3bpjNdTjCNXoUGehEiDix+oaXMz7aBM/bimke2oUl07qT7cUnexF/qNixbfk/ftpQnv3If3GW3CqAHQqKgEifqq4vI7H3l3Lnvwqzj62B2cd24MApyan65hxAAAgAElEQVT/yX9UfPs1ec/+m7B+BunX34QzNNTuSNLKVAJE/NDa7cU8MX8dlqVj/+XAAuMTiBgylLQrrsIZEmJ3HGkDKgEifubrdbm89KlJcmwY100ZTLJm/8tv1GdnEZKeQbjRn3Cjv91xpA1p7E/ET9TUNfL60q3MW7CJ9MRIbp42VAVAfqf000Xs/uufqV631u4o0g40EiDiBzbtKmHewk2UVNRzzKBULplkaO1/+Z2ShR9R9O7bRI4YSfiAgXbHkXagEiDSie27+E9CdAh3zhxOv65a4U1+zbIsSj76gOL33yNq9BhSL7sCR4BKoj9QCRDppDwei5cXb+Hz1dmMHpjCrFMNwkP1Ky+/V7d9G8Xvv0f02ONIueQyHDpKxG/oFUGkE3K5PTy3cDPfbsjj5OEZzJjQF6cW/5EDCOvTl/SbbiV84CAVAD+j/22RTqaiuoE5r63m2w15nHFMdy5UAZD9sCyLwrffpHb7NgAiMo9SAfBDGgkQ6URyiqqZ+946CkprueLMgRyTmWp3JPFBlsdDwcsvUr78cxyBAYT17mN3JLGJSoBIJ5FVUMU9r6zC4YDrpwxmcG+d5EV+z/J4yH/+WSq++Yr4088k4Zzz7I4kNlIJEOkE8kpquPfVVQQFOvnTrBE6/a/sl+V2k/fsv6lc8S0JZ08m/qxzdKIoP6cSINKBeSyLZauyefuL7TgdDu6YOVwFQA7KcjWSeN75xJ9+pt1RxAeoBIh0UKWV9Tz/8WbW7Simf7dYLj6tP6nx4XbHEh/kaWzEU1dLYFQ0aVdeowmA8guVAJEOaE9+Jfe/tprqOhfTxvVh4tFdNawr++VpbCD3ybk0FhXR7S9/wxkUZHck8SEqASIdzLcb8nj5U5MAp5P/uXgEvbvE2B1JfJSnvp6cJx6jZsN6kmddogIgv6MSINJBuNwenv94M9+szyMtIZybpw4hUfv/5QA89fVkP/YwteZmUi65nJjjjrc7kvgglQCRDiC3uJpnPtzIrrxKTju6G+ed2IvAAO3XlQMreO1las3NpF5+BdFjxtodR3yUSoCIj/t2Qx7PLdxMSJCTq84ZxNEDUuyOJB1A4uQpRA4dTuTQYXZHER+mEiDiozyWxZIfsnhr2Ta6pUTxx7MHkhKn2f9yYO6qKkoXLyLh7MkExsaqAEiLVAJEfNTzH2/mq7W5DOgexzXnZhIRqkldcmCuygqyH7yfhtxcIocNJ7RHT7sjSQegEiDiYyzL4rVPTb5am8v44enMnNBPh//JQbnKy8l64D4aCwvocv1NKgDiNZUAER9SVdvI8x9vZtWWQkb1T2bGKX1VAOSgXGWlZM25j8aSYtJvvIXw/gPsjiQdiEqAiI8oKqtl7vz1ZBdWceHE/owfkobTqQIgB9dYUoq7rpaMm2cT1ref3XGkg1EJEPEBKzbm8+KizbjcFledk8nEY3tRWFhpdyzxYe7aWgLCwgjr1Yued9+HMyjY7kjSAelAYxGbLf8ph6c+2EBKXDh/u3QUw/sl2R1JfFxDfj67//o/lC37DEAFQA6bRgJEbLRyUz7Pf7yZfl1jufH8wYSF6FdSDq4hN4e9c+4Dt5vQ3r3tjiMdnF5xRGyyclM+T72/gd5dorl56hBCggPsjiQ+rj47i6w594EDMm67g5D0DLsjSQenEiBigy17y3jqgw10TYnklulDVQCkRe7qarLm3AvOALrOvp3gtC52R5JOQCVApJ1tyy7nsXfWEh8Vym0zhmkXgHglICKCxClTCetrEJyipaOldejVR6QdZRVU8dCbPxEeEsAt04doFUBpUe32bVguF+FGf2KOO8HuONLJ6OgAkXaybkcx/3j5R4ICncy+YBhpCRF2RxIfV7t1C1kPzqHw9VexPB6740gnpJEAkXawemshT87fQGp8GNdNGUxybJjdkcTH1WzeRPajDxEYH0/6jTfjcOozm7Q+lQCRNrb0xyxeWbyFLokRzJ4xjOhwHdMtB1e9YT05jz9CUFIyGbfeTmBMjN2RpJNSCRBpQ8tWZ/PK4i0M6B7HdecdpUmA4pXK71cQnJpK+i23ERgVbXcc6cT0iiTSBizLYtHKvbz9+Xb6d4vlpqlDCArUcK4cnOVy4QgMJGXWJXjq6wgI17wRaVt6VRJpZfWNbp6cv543l23jqF7xXD9lsAqAtKjyh5Xs/vtduMrKcAQEqABIu9BIgEgrqmtw8cT89azfUcI5x/XkrGN74NSpgKUFFd99Q968Zwjr0xdnaIjdccSPqASItJLaehcPv/UTW7PKmTXRYNywdLsjSQdQ/tWX5L/wLGFGf9KvvwlniEqAtB+VAJFWUFJRx+PvrmN3XiV/PHsgYwam2h1JOoDK71eS//w8wgdl0uWa61UApN2pBIgcoZyiah568ydKK+u58pxBHD1AS7qKd8L7DyD2lAkkTpmq0wGLLTRbSeQIbM8pZ87rq6lrcHHL9CEqAOKVyh9/wHK5CIiKIvmCmSoAYhuVAJHDtGl3Kfe/thrLgtkXDGNgj3i7I0kHUPzRB+Q++Thly5baHUVEuwNEDoe5p5TH3llLbGQId1w4nLgo7cuVg7Msi+IP5lPy4ftEjTmG2PGn2B1JRCVA5FBt3l3Kw2//RExEMLfPGKYCIC2yLIuid9+m9OMFRB97PCn/danOBSA+QSVA5BBs2FnCQ2/+RGJMKLdfOIz46FC7I0kH4Cotofzzz4g5cRzJM2epAIjPUAkQ8dL6HcU8+s46kuLC+J9ZI4gMC7I7kvg4y7JwOBwExSfQ/a6/E5iYiEOLR4kPUR0V8cLa7cU89u46EqJDuH3GMBUAaZHl8VDw0vOULPwIgKCkJBUA8TkqASIt+HxNNo+9s5b46FDunKlJgNIyy+Mh//l5lC//Ak9dnd1xRA5IuwNEDuLTlXt4/bNt9OoSzQ3nDyY6XMdzy8FZbjd5zz5D5YrvSDjnXBLOOsfuSCIHpBIgcgA/bC7gjc+2MbBHHDdPG0KAJnNJCyzLIu/fT1H5/UoSzzuf+NPPtDuSyEGpBIjsx/ebC3jmww10S4ni+vMGqwCIVxwOB+EDBhHasxdxp55mdxyRFqkEiPzGio35PP3hBrqnRHHTtCGEBAfYHUl8nKexgfq9WYT16kXMCSfaHUfEa/p4I7KP7zbk8e+PNtInPYbbZgzTHABpkae+npxHHyHrgXtxlZfbHUfkkKgEiDTbmlXGvAWbSE+K4IbzBxMWooEyOThPXR3Zjz5EzeaNJF84i8CYGLsjiRwSvcqJ0HQ64EffXktkWBC3zRhGRKjWAZCDc9fWkv3Ig9Tt2E7qH/5I9Ohj7I4kcshUAsTv5RZXc88rq5rPBjhUBUC8Uv75Z9Tt3EHaH68mauQou+OIHBaVAPFrpZX1PPjGGizL4s6Zw0lPirQ7knQQcRMnET5gIKE9etodReSwaU6A+K26BhePvbOWippGbpk+lIxkFQA5OFdFBdmPP0JjcTEOp1MFQDo8jQSIX/JYFo+/u47deZVcPTmTnmnRdkcSH+cqKyPrwftoLCqisbCAoIQEuyOJHDGVAPE7jS43z3y0iY27Spk5oR8j+yfbHUl8XGNpKVlz7sVVVkr6DTcT3n+A3ZFEWoVKgPgVy7J4ZfEWfthcwDnH9WT88HS7I4mPaywpJuv+e3FXVpBx062E9e1ndySRVqMSIH7DY1m8sXQby3/K5dRRXTnnOO3PlZY5Q0IJTEgg9YorCevV2+44Iq1KJUD8gmVZvLzI5PM1OYzsn8y0cX3sjiQ+rrGokICYGAIiIsi49XYcDofdkURa3SEfHWAYhnagSofz2apsPl+Tw6mjunL1OYNwOvWCLgdWn5PDnrv/Qf6LzwOoAEin5dVIgGEYMcBjwFTAA0QYhnEWMNI0zb+2YT6RI/bTtiJeXbyFfhkxTBvfRy/oclD12VlkzbkPHBB/2hl2xxFpU96OBDwJ1AH9gIbmbSuAGW0RSqS17Mmv5Mn315OaEM715w/GqQIgB1G3Zzd7778HApx0vf2/CUnXxFHp3LwtAacA15mmuRewAEzTLABS2iqYyJGqrmvk8XfXER4SyM1Th2g5YDkoy+0m96kncAaH0PX2PxGcmmZ3JJE25+3EwAogHsj7eYNhGF2B/LYIJXKkPB6LJ95bT0lFPbdfOIzE2DC7I4mPcwQEkHblNQSEhxOUmGR3HJF24e1IwLPAW4ZhHA84DcMYBTwHPNVmyUQOk8dj8a/317NpdynTx/ehX9dYuyOJD6vZYlK84EMAQrt1VwEQv+LtSMDdQD0wDwgFXqWpADzk7QMZhtEPeAFIAIqBi03T3Lqf600D/gI4aNr1cIppmhpxEK9YlsVrS7byg1nI2cf2YMKornZHEh9Ws2kj2Y89TFB8AnEnT8AZGmp3JJF25W0JSDBN8wHggX03GoaRCBR5eR//AuaapvmyYRgX0VQixv/m/kYCfwPGm6aZ13xUQr2X9y/C0h+zWLoqi3HD0rUYkBxU6arVZD/6EEHJKWTccpsKgPglb3cH7DjA9i3e3Lh5bYHhwGvNm14DhhuG8dtxt5uBOaZp5gGYpllummadlxnFz63YmM+rS7YyqEccF07oq0MB5YCq1qxm0z/uITg1ja6z7yAwJsbuSCK28HYk4HevpoZhRNK0ZoA3ugLZpmm6AUzTdBuGkdO8vXCf6w0EdhqGsRyIBN4F/mGapuXl45CQoNPBtrWkpCi7I/zO12tzmLdgIwN6xPO/V48lKDDA7khHzBef587CCrKI6NWTQX/9M4GRes1oS/o59m0HLQGGYeykab98mGEYvx0NSATeaeU8AcBgYAIQDHwC7AFe9PYOiour8Hi87gxyiJKSoigsrLQ7xq/syKngoddW0zU5iqvPGURZaY3dkY6YLz7PnYGrvJzAmBgcg4Yz+PjjKCqpgVo9z21FP8dty+l0HPEH35ZGAv5A0yjAB8AV+2y3gHzTNDd4+Th7gXTDMAKaRwECgC7N2/e1B3jbNM16oN4wjPeBozmEEiD+paC0hkfe/onQ4ACuPTeTyDCtBSD7V/Ht1+S/9AIZN99GWN++OAI6/miRyJE6aAkwTXMpgGEYqaZpVhzug5imWWAYxhqaVhh8ufnv1aZpFv7mqq8CpxuG8VJztpOBtw/3caVzKyqvZc7ra2hwefjzrBHER2til+xf+VfLyX/hOcKM/oR062Z3HBGf4dWcANM0KwzDyASOp2k3gGOff/u7l491FfCCYRh3AaXAxQCGYSwE7jJN8wfgdWAksJGm+QaLaDosUeRXisprufeV1VTVNXL9eUeRnqT9urJ/Zcs+o+CVFwkflEmXa2/AGRxsdyQRn+HtCYQup+kEQktp2l+/mKZP6R96+0CmaW4GRu9n++n7XPYAtzT/Edkvl9vDE++tp7K2gVumDdViQHJANZs3UfDKi0QMHkLa1dfiDFIBENmXt4cI3gmcbprmWUBt89/TgOo2SyZyAK8t3cquvEouP2OgCoAcVJjRn+SLL6HLNderAIjsh7clIMU0zc+bL3sMw3ACC4DJbZJK5AC+XJvDslXZnDwig1H9k+2OIz6qdMmnNBQU4HA4iD3hJByB3h4NLeJfvC0BWYZhdG++vBU4AxgDNLZJKpH9WLWlkOc/3syA7nFMH9/H7jjigyzLomj+uxS+/irlXyyzO46Iz/O2Hj8AZAK7gf8D3gKC0L57aSclFXU889FG0hMjuXpyJoEB3vZX8ReWZVH0zluUfrKQ6ONOIHHKVLsjifg8b48OmLfP5Y8Mw4gDQkzTLG+zZCLNLMvi1SVbcbstrQUg+2VZFoVvvEbZkk+JOWk8yRdehMOpoijSksP6LWlezz/QMIy7WzmPyK9YlsXLi7ewakvTWQFT4sPtjiQ+yGpooG7HNmJPmUDyzFkqACJeanEkwDCM/wKG0jQX4GkgAvgzTcf9f9Om6cTvffTNLpatyuakYemcfkz3lm8gfsXyeLBcLpwhIWTcegeO4GCdOErkELR07oD7gFk0vdnPoGky4DHAj8DxpmmuafOE4reWrc7mvS93Mqp/MrNO7acXd/kVy+Mh/7l5NJaWkHHTrThDQuyOJNLhtDQScAFwgmmaWw3DGABsAGaYpvlG20cTf5ZXUsNrS7YwoHscV5w1UAVAfsVyu8mb9zSVK1eQcM65OgRQ5DC1tOMs1jTNrQCmaW4CalQApK25PR6eXbCJoEAnfzhzoI4EkF+xXC5yn36SypUrSJwyjYSzzrE7kkiH1VJ9dhiG0ZX/nCvA9ZuvMU1zT1uFE//0yYo9bMsu57LTBxAXpSFe+bX8l1+k6scfSJo+g7gJE+2OI9KhtVQCIoBd7POmT9NaAT+zAJ2PU1pNSUUdH36zi8G9Ezj2qFS744gPiptwKmG9ehNzwol2RxHp8FoaZw0Cgpv/3t8fLcYtrcbt8fDvjzbi8cCFEzQRUP7DU19P2fLPsSyLkPQMFQCRVnLQkQDTNN3tFUTk7c+3s3lPGbMmGiTHhtkdR3yEp66W7EcfpnbrFkJ79CS0mw4VFWktmlIrPuGrtbksWrmXE4Z0YdywdLvjiI9w19SQ/ciD1O3cQeoVV6oAiLQylQCxXW5xNS98spk+GTHMnNDX7jjiI9zV1WQ9NIf6vXtIu/JqokaMsjuSSKejEiC2anS5eXL+ekKDA7j6nEyCAjXPVJrU7dxOQ042Xa6+jsihw+yOI9IpeV0CDMMIBEYB6aZpvm0YRhiAaZq1bRVOOr93vthBVmE110zO1OGAAjStBOhwOonIHEzPu+8nMCbG7kginZZXq7AYhjEI2Ay8BDzfvPlk4Nm2iSX+YHtOOZ+tymLMoBRG9k+2O474AFdZGXv+969UrV4FoAIg0sa8XYrtSeD/TNPsAzQ2b/scOL4tQknnl1dSw2PvrCMyLIjp4zUPQKCxpIS9999NQ0EBznCdLVKkPXhbAo4CXmi+bAGYplkF6DdVDll9o5uH3/wJt9vDrRcMIyZCy034u8biIrLuvxt3RQUZN88m3OhvdyQRv+BtCdgN/GpmjmEYI4HtrZ5IOjWPx+L5jzdTUFbLlWcPIj0xwu5IYjNXZQV7770bd3U1GbfcRlgfjQyJtBdvJwbeBSwwDOMJINgwjNuAa4Gr2yyZdDoey+LFRZtZsTGfycf1JLNXgt2RxAcEREYRfcxYIkeM1DoAIu3Mq5EA0zQ/AM4GugJfAwYw3TTNj9swm3Qy7y3fwfKfcjl5RAZnHdvD7jhis/qcbBrycnE4HCSeO0UFQMQGXo0EGIYRZ5rm98D3bZxHOqmNu0pY+N1uRhhJXHhKX50XwM/V791L1oP3ERifQLc//1U/DyI28XZ3QLZhGEuAV4APtDaAHIqSijqeeG89qfHh/Ndp/fWC7+fqdu8i68H7cQYHk3bFVfp5ELGRtxMDewJLgJuBPMMwXjIMY5JhGFreTQ7K7fHwr/c30ODycN15RxEZFmR3JLFR7Y4dZD1wH86QUDJu+2+CU3W6aBE7eTUSYJpmPvAo8KhhGL2AC4E5QCKQ0nbxpKN7/6tdbMsu5/IzBpCWoCMB/F3Jgg8IiIggY/YdBCUk2h1HxO8dzrkDYpr/RAHVrRtHOpO124tZ8M0uxgxKYWymPvH5M8uycDgcpF1xJe7aOoLi4uyOJCJ4PzGwHzCDphGAGOAt4ALTNL9pw2zSgdXUNTJvwUa6JEZw8URD+339WPXGDZQu+pguV1+HMzQMZ2iY3ZFEpJm3IwHfA+8BNwBLTNN0t10k6eg8lsXc99ZTVdvILdOGEhqsk1X6q+p1a8mZ+yhBKal4GhtwhobaHUlE9uHtq3OKaZp1bZpEOo33lu9g0+5SZk7oR/fUKLvjiE2q1qwm919zCe6STsYttxEQGWl3JBH5jQOWAMMwZpim+Vrzl9MMw9jv9UzTfLEtgknH9NmqLBZ8u5uxmamMH55udxyxSdWa1eQ8+TghXbuRcfNsAiI0KVTEFx1sJOAS4OcScMUBrmMBKgECQFZBFa8v3Upmz3guO32A5gH4seDUNCKHDCXlkssJ0BkBRXzWAUuAaZoT97msUwbLQTW63Dwxfz3hoUH84cyBOJ0qAP6odsd2Qnv2Ijg1lS7XXG93HBFpgVeLBRmGsd/lgg3D+K5140hHZFkWLy4yySup4fIzBhCtUwP7pfLlX7D37v+jfPnndkcRES95OzHwQCf37tdaQaTj+mxVNl+vy2PCyK4cpTMD+qWyZUspeOUlwjOPIvqYY+2OIyJeOmgJMAzj2eaLwftc/lkPYFNbhJKOY1t2OW98to0B3eOYfnIfu+OIDUoXL6LwjdeIGDqMtCuvwRmkpaFFOoqWRgKyD3DZAn4E3mj1RNJh1Na7eOK9dYSHBjbNA9BEQL/TUFhA4dtvEjliZNPJgAK1JoRIR3LQ31jTNP8CTfv+TdNc0D6RpKP48OtdlFU1cMeFw4iLCrE7jtggOCmZrrf/N6E9euII0PnERDqag60TcKxpml83f1lpGMYJ+7ueaZrL2ySZ+LT1O4pZ9P0exgxMweimdeD9iWVZFL//HiFd0ok6ejRhvbUbSKSjOthIwDz+MyHwlQNcxwK6tWoi8Xm78yp5Yv56kmLDmDVx/4tISedkWRZFb79J6aKPiTlxHFFHj7Y7kogcgYOtE9B/n8td2yeO+LqaukYefWctocEB3DR1CGEh2gfsLyzLovCNVylbspiYceNJnnGR3ZFE5Ah5tU7AbxmGcbxhGGNbO4z4vucXbKS0sp6rzskkNV4rwfkLy7IoeOUlypYsJnbCRJIvnIXDeVgvHyLiQ7xdLOhzwzCOa748G3gXeMcwjDvaMpz4lj35lSz6dhfjh6fTr2us3XGknTlDQ4k77XSSpl2gJaFFOglvx3KPAn5eHfBK4CSgEvgSuLf1Y4mvcXs8zFuwiaiIYM45rqfdcaSdWG43rtISghKTSJwyFUAFQKQT8XY8zwl4DMPoBQSaprnBNM09QHzbRRNfMv/LnewtqOLSMwcRFa5lgf2B5XKR9++n2POP/8VdVYXD4VABEOlkvB0J+AZ4GOgCvAfQXAiK2yiX+JD1O4pZ8O1uxgxKYfzIrhQVVdkdSdqY5XKR+9STVK3+kcSp0wmIjLQ7koi0AW9HAi4B6gAT+GvztoHAY22QSXxIXYOL5z/ZTEpcGP81sb8+CfoBT2MDOU88RtXqH0m6YCbxEyfZHUlE2ohXIwGmaRYCt/9m20fAR20RSnzHG59to6SinjtnDickWCvC+YOSBR9RvfYnki+6mNiTxtsdR0TakFclwDCMQOC/gVlAOk3nEXgJuMc0zca2iyd22p1XyfI1OYzT0QB+JX7SGYT26Enk0GF2RxGRNubt7oB7gTOAm4Cjm/+eBNzTRrnEZm6Ph3eWbyc4KIBzj+9ldxxpY566WgpefQl3bS3OkBAVABE/4e3EwGnAMNM0i5q/3mAYxvfAGuDWNkkmtnp1yVbW7yhh2rg+RIbp1LCdmbumhuxHHqRu5w4ihgwjYlCm3ZFEpJ14WwICAM9vtnk4zBUHxbftzK1g2apsxg1P57TROjVEZ+auribroTnU791D2lXXqgCI+BlvS8DbwAeGYfwV2AN0B+5q3i6dSE2di8ffXUdMRLB2A3Ry7spKsh68n4bcHLpccz2RQ4baHUlE2pm3n+RvA5bTdGbBDcAzwNfN26WTsCyLec3nBrjy7EHaDdDJeerr8DQ00OW6G1UARPyUt4cI1gN/av4jndS3G/JYvbWIKSf2on/3OLvjSBtxV1XhDA8nKDGJHn//B44AHfop4q8OWgIMw+hL06f/TGAVcFnzcsHSybjcHl5bspXuqVGaB9CJNZaUkPXAvUQOHkrS9BkqACJ+rqXdAY/TtCbAJUARTUsHSye09McsqutcTDmhFwE6RWyn1FhUSNZ9d+OuqCBy5Ci744iID2hpd8AIoKtpmrWGYSwDNrdDJmlnO3MreOeLHQzpncCgnjonVGfUUFBA1px78dTVknHLbYT21KRPEWl5JCDYNM1aANM0K4Gwto8k7cljWbz4iUlEaCCXnTFA5wbohCyXi+yH7sfTUE/G7DtUAETkFy2NBIQYhnHXPl+H/eZrTNP8e+vHkvbyzbo8dudXcsmk/jpFcCflCAwkeeYsAuPiCUnPsDuOiPiQlkrAm0Dffb5++zdfW62eSNpNeVU9by7bRu/0aI4bnGZ3HGll9Xv3UJ+TTfToY4jIHGx3HBHxQQctAaZpzmqvINL+Xlu6ldp6F/81sT9O7QboVOp27SLrwftxhoUSOWwEzmCN8ojI72kauJ/6YXMBKzcVcNrobmQkR9odR1pR7fZtZD1wL86wULrOvlMFQEQOyNtlg6UTKSit4bmPN5ORFMGZY3vYHUdaUe3WLWQ/8iABUVFkzL6ToIQEuyOJiA9TCfAzLreHue+tx+OxuPa8owgJ0mIxnUnt1i0ExMSSMfsOguK06qOIHJxKgJ9Z/MNe9hZUcc3kTFLiwu2OI63E09CAMziYuElnEDv+ZJyhOppXRFrm9ZwAwzDGGYbxlGEY85u/Hm4YxoltF01aW0FpDfO/3MmA7nGM7J9sdxxpJVVrf2Lnn26nPjsLh8OhAiAiXvOqBBiGcQ1N5xDYC4xr3twA/KONckkbePuLHTgccNnpA+yOIq2kavUqcuY+SmB0DIExsXbHEZEOxtuRgFuBU0zT/D/A07xtE6B3kw5iW1Y5P2wu4NRRXUmICbU7jrSCyh9WkvOvuYR2707G7NsJiNRRHiJyaLydExAF7G6+/PMCQYH8//buPD7q6t7/+GsmG9mBEEI2NpEDiogIbgV3rUutaK2WWte69XrtVSvVen/22tveqqjVal2o2uqtlmtr61pbFdEqakUFRQEPO2QjCSF7QjKZ+f7+mMFGGsgEMvOd5f18PHhIvvlmvu8cYr6fOed8zwn2BkiM6/EH+O1fVzM0J51TDx/jdhwZBB2rV1Gz4CGG7DeB0rxPwrAAACAASURBVP+4npRMDQGIyMCF2xOwBLhhl2NXA38f3DgSCa8s3UJNQwcXnTKJzAzNBU0EQyZMYPipp1N27Q9UAIjIXgv3jnAN8JIx5nIg1xizkmAvwGkRSyaDoq3Tx0vvbmbahBEcPGGE23FkH7V+sJSsyQeQkpPDiLPPcTuOiMS5sHoCrLVVwHTgIuBC4EpghrW2JoLZZBC8sGQjXT4/Zx2tnePiXePiRdQseJDtf33J7SgikiDC7hu21jrAO6E/Egc2b21l8bIqZh1UTLmWBo5rja++Qv0fFpI97RAK5nzD7TgikiDCKgKMMRvZzY6B1lq9xYxBjuPw1GtryMlK49zjJ7gdR/bB9pdfYtufnyFnxkyKL7sST6rmdYjI4Aj3t8llu3xcTHCewMLBjSODZcmKGtZVNXPBVw05mWlux5G9FNjRSfNbfyf38CMYdenleFK0zLOIDJ6wigBr7eu7HjPGvA68DNwbzmsYYyYCTwAFQANwobV27W7ONcBy4EFr7a5PJUg/Gpp38NRraxhfksfRBxe7HUf2guM4OIEA3iGZlP/oP0nJzcPj1aafIjK49uW3SicwkKGAh4EHrLUTgQeABX2dZIxJCX3uuX3IltT++OY6Ag5cccYBpOjGEXccx2HT4/9L7RO/xQkESM0fqgJARCIi3DkBP97lUBZwOvBqmF8/kuDTBSeFDi0EfmWMKbTW1u9y+k3AS0BO6I8MwNLVtSxdXceph49mpDYIijuO41C/8CmaFi9i6PEngMfjdiQRSWDhzgnYf5eP2wm+m388zK8vB6qstX4Aa63fGFMdOv5FEWCMORj4KsH9CW4J87W/pKAgeeuGHd09LHx9LRNHD+WKbxxMakpk3j0WFuZG5HWTnRMIsP7hR2havIiSM89g7CUX4VEREFH6WY48tXFs67cICHXPvwb8wVq7I1JBjDFpwK+BS0JFwl69TkNDG4FAnw8yJLzFyyppbuvmijMOpHF7e0SuUViYS319a0ReO9nVPvU7mt94neGnfY2xl1zEtm1tbkdKaPpZjjy1cWR5vZ59fuPb71vF0Lv3+/exAKgASkMFxc7CoiR0fKdiYD/gZWPMJuBa4HJjzK/34bpJo7Orhxff3cS44lwmjdZucvEod/qhFMw5m4KzvqEeABGJinD7i/9ijNnrJYKttXXAx8Dc0KG5wPLe8wGstVustSOstWOttWMJPnXwiLX2ir29bjJ57u2NNLd1M/fEibqBxBGnp4f2lZ8BkDX5AAq+9nX9+4lI1IQ7J8AL/NkYs4Tgu/cv+tuttZeG+RpXAU+EJhk2Elx+GGPMy8CPrbUfhp1avmRjTQuLl1Vy1JRRTCjNdzuOhMnp6aFmwUO0fbyMMbf+jIzSUrcjiUiSCbcIWAvcuS8XstZ+Dhzex/E+existbfuy/WSheM4LFy0luwhqZynlQHjRsDXTc1DD9C+4hMK556vAkBEXLHHIsAYM9dau9Bau1cz9SXy3l9Vy7qqZs4/aSK5Welux5EwBLq6qH7wfjpWfsbICy5m6DHHuh1JRJJUf3MC+lzQR2JDZ1cPv1+0ltFFORwzrcTtOBKmtk+W07FqJUUXf1cFgIi4qr/hAM1QimHPL9lIW6eP686N3JoAMvjyDjuCjNIyMkrL3I4iIkmuvyIgxRhzHHsoBqy1iwc3koSjsbWLxcuqOPyAIsYV57kdR/rh72inZsFDjJhzNkPGjVcBICIxob8iIAN4jN0XAQ4D2z9ABsmL72zEHwgwZ9Y4t6NIP/xtbVTecxddlRX0NDW5HUdE5Av9FQHt1lrd5GNMfVMnb35czfHTSykarv0BYllPawtVv7iT7poaSq6+hpyp09yOJCLyhXAfEZQYsnhZJR7gq4eNdjuK7IG/tZXKO+/AV19HyTXXkn3gFLcjiYh8iSYGxpn6pk5e+6CSIw4cReHQTLfjyB54MzPJKC9n5Le/Q9akyW7HERH5F3ssAqy12v4pxjz39ga8Xg9nH61Rmljl296AJzWN1Lw8ii+/yu04IiK7pefK4khdUydLV9dx/PRSCvKHuB1H+uCrr6di/m3UPPQrHCc5d7MUkfihIiCOPPfWBlK8Hk6eWe52FOlDd+1WKubfRqCjk8Lzvq2NgEQk5mliYJzYUtvK+6trOWlGOcPz1AsQa7qqq6m8ez74/ZTPu5GMck3aFJHYpyIgDgQCDo++tIrM9FROO2KM23FkF47jUPfkE+AEKJt3oxYCEpG4oSIgDqxY30BlfTuXn3EAednaJCjWeDweRl1+Fc6OTtKLtYeDiMQPzQmIcY7j8Pw7Gxmel8HMSSPdjiO97Ni0kdrfPY4TCJA2bJgKABGJO+oJiHFvr6hh89ZWLjl1kjYJiiGd69dRde/deLOz8be2kJo/1O1IIiIDpiIghjW2drFw0Vr2L8vnK1OL3Y4jIZ1r11B57y9IzcujbN6NKgBEJG6pCIhhz769ge4eP5eePhmvHjeLCR2fr6bqvntIHT6c8htuJHXoMLcjiYjsNRUBMWpjTQtLVtRwwvQyioZpk6CY4fWSUVpGyb//B6n5+W6nERHZJxpkjlHPvLmenMw0ztLywDHB17ANgKyJhvKbb1EBICIJQUVADFpf3czqzY2cdsQYsoaos8Ztrcs+YuPNN9K69H0ArQQoIglDd5gY9Jd3N5OZkcKxh+iRM7e1frCUmkceZsjYcWRNOcjtOCIig0o9ATHms40NfLxuGyfNKGdIumo0N7W89y41v36IzP0mUHb9DaRkaW6GiCQW3WViiOM4PPPGeoblZmh5YJd111Sz9TePkGkmUXrNtXgzMtyOJCIy6FQExJBPNzSwpa6N75w8kfS0FLfjJLX04hKKr/w3sg+aqgJARBKWhgNiRCDg8Mc31jM8L4NZB2lhILc0vbGYzrVrAcidMVMFgIgkNBUBMWL15kaqtrVzzrH7qRfAJdtf+St1T/0vTW+94XYUEZGo0HBAjHj3s62kpniZNmGE21GSUsNfXqTh2T+RM+MwRl10qdtxRESiQkVADKisb+P9VbUcN71UTwREmeM4NLzwHNtffJ7cw49k1KWX4UlRT4yIJAfdcVzmOA6/f20NaWlezjhqrNtxko/j0F1TQ95Rsyi6+FI8Xo2QiUjyUBHgso9sPZ9vaWLO7HHkZae7HSdpOI5DoLODlKxsii+7ArxeFQAiknT0W89FgYDD04vXUVaYrXUBosgJBKhf+CQVP/8Z/o4OPKmpKgBEJCnpN5+LPvi8joaWHZx+5FhSU/RPEQ1OIEDdk0/QtPh1sg8+GG9mptuRRERco+EAl3R29fDMm+spGZHNoabQ7ThJwQkEqH38MVrefYfhp59BwZyztRmQiCQ1vf10yYvvbqKhZQcXnDxRvQBR0vD8s7S8+w4FZ57FiLO+oQJARJKeegJc0NnVwxvLqjj8gCLM6GFux0kaQ48/kdThBQw95li3o4iIxAS9BXXBeyu30uXzc9whpW5HSXgBn4/tf3sZp6eH1Px8FQAiIr2oJyDKfD0BXl1awbjiXPYvy3c7TkIL+LqpefBXtH+6goyyMrKnTHU7kohITFFPQJR98HktdU2dnDxztMakIyjQ1UX1fb+k/bNPGXnhxSoARET6oJ6AKPIHArz47maKC7KYOXmk23ESVmDHDqruv5fONZaii79L/ldmuR1JRCQmqScgij6y9dRu7+Cs2ePxqhcgYnz1dXRVVDDqsitVAIiI7IF6AqKkxx/g5fc2MyJ/CNO1LkBEBHw+vGlpZJSPZtzt80nJynY7kohITFNPQJS8t3IrW+ramDN7nHoBIsDf1kbFbT+j8bVXAFQAiIiEQT0BUfL2JzWMHJrJkQeOcjtKwulpbaHy7jvxba0hrUjtKyISLvUERMGW2lbWVTUz++BiPREwyHqam6i883Z8dbWUXHMtOVMPdjuSiEjcUE9AFLyxvIr0VC9HH1zidpSEEvB1U3nnHfgat1P6/evImjTZ7UgiInFFRUCEtXZ0s3R1LdNNIblZ6W7HSSjetHSGnvRVMkpKyNx/ottxRETijoqACHvz42o6u/yccthot6MkDF99Pb7tDWSZSVoGWERkH2hOQAR1dft5Y1klk0YPZXRRrttxEkJ37VYq5t/G1sd+TcDnczuOiEhcUxEQQW9+XEVTWzdnzhrndpSE0FVdTcX823F8PkqvuQ5vWprbkURE4pqGAyLEHwjwt/e3MLEsn4nlQ92OE/e6qiqpvGs+eKBs3k1klGoHRhGRfaWegAhZvbmR5vZuTpxRrscCB0HzW3+HFC/lP/yRCgARkUGinoAIee2DSnIy0zh4QoHbUeKa4zh4PB4Kz5vLsFNOI23YMLcjiYgkDPUERMCW2lY+3dDA8dNLSUtNcTtO3Opct5YtP70VX2MjHq9XBYCIyCBTERABi5dVkpbq5cQZ5W5HiVsdayyV99xNoGsHOAG344iIJCQNBwyytk4f76+u47BJI8nJ1Oz1vdGxehVV999L2vACym64kdShmlgpIhIJKgIG2fNLNtLt83P8oWVuR4lLHWssVffdQ9rIIsqun0dqfr7bkUREEpaKgEHU2NrF3z+u5sgDRzGuOM/tOHEpo6SUnENnMPK8b5OSqwWWREQiSXMCBtHfP66ixx/gtCPGuB0l7nSssQR8PlJycii+7EoVACIiUaAiYJB0dvXw2ocVTN2vgJIR2W7HiSutS9+n8q472P7SC25HERFJKhoOGCSvf1RJZ5efM44a63aUuNLy3jts/c2jZO4/keGnnuZ2HBGRpKIiYBB0+fy8+kEFU8YPZ79STWQLV/OSt6h94rdkmkmUXnMt3owMtyOJiCQVFQGDYMmKGto6fZyuuQBh87e3U//Hp8k64EBKrv4+3vR0tyOJiCQdFQH7yHEc3v6kmvKROZjRWtEuXCnZ2ZTfeDNphYV401QAiIi4QRMD99G6qma21LVx9MElbkeJC9v/9jINoQmAGSWlKgBERFykImAfOI7DwkVryctO56gpo9yOE/MaXnqBbc/8ge6qSpyAlgIWEXGbioB9YLc0sWlrK2ccNZbMDI2s7I7jOGx77s80PPdnco88ilGXX4XHqx89ERG36c61D17/qJK8rDRmHVTsdpSYtu3Pz9D417+QN+toii68WAWAiEiM0G/jvdTt87Ny03am7V9IRrq2C96T9KJR5B93ggoAEZEYo56AvbRsTT07uv0cNnmk21FikhMI0F1TTUZpGfmzZpM/a7bbkUREZBd6W7aXlnxaw7DcDCbpscB/4QQC1P7ucbb89Fa6a7e6HUdERHZDRcBeqNrWzqpNjcyeWozX63E7TkxxAgFqf/sYLW+/xbBTTiNtZJHbkUREZDc0HLAXXv+wghSvh2MPKXU7Skxx/H62PvZrWpe+T8GZZ1FwxpluRxIRkT1QETBA9U2dvL2ihq8cNIqhOVrrvreWd5bQuvR9RnzjXG0GJCISB1QEDNDS1bX4Aw6nHznW7SgxJ2/WbFILCsg+cIrbUUREJAyaEzAAjuPw5vJqJpTlUzg00+04MSHQ3c3Wx3+Db1s9Hq9XBYCISBxRETAAn29upKFlB8dO0z4BAIGuLqrv/yUt77xN5/p1bscREZEB0nDAAPxtaQU5mWlMn1jodhTXBXbsoOq+e+hcu4ZRl1xG3uFHuh1JREQGKGpFgDFmIvAEUAA0ABdaa9fucs4twLcAP+ADbrbWvhKtjHtS19TJZxsbOO2IMQxJT+7ayd/ZSdW9d7Nj4wZGXXYleYcf4XYkERHZC9EcDngYeMBaOxF4AFjQxzlLgZnW2qnApcDTxpiYGHxf9GEFHjwcp8cCwQnuAFh85fdUAIiIxLGovKU1xowEpgMnhQ4tBH5ljCm01tbvPG+Xd/0rAA/BnoPKaOTcHV+Pn3c/3cohE0cwPG+Im1Fc5W9rw5+XTkpWNuU33qx9AERE4ly0+rXLgSprrR/AWus3xlSHjtfv5msuBNZbawdUABQU5OxT0L68s6Kajq4evjZ7PwoLcwf99eNBd1MzK++ZT+OoIibffJPbcZJCsv6sRZPaOPLUxrEtJge3jTHHAD/lnz0HYWtoaCMQcAY1z9OvWkYOzaRs+BDq61sH9bXjQU9TE5V3z8fXsI1x370kKdsg2goLc9XOEaY2jjy1cWR5vZ59fuMbrf7cCqDUGJMCEPpvSej4lxhjjgSeBOZYa22U8u1WXVMnG2taOOaQElKSsPvbt307FXfehm97A6X/cT1DD57qdiQRERkkUbmrWWvrgI+BuaFDc4HlvecDABhjZgJPA+dYa5dFI1t/loUiHpqEjwU6jkPNww/gb26m7LobyDKT3I4kIiKDKJrDAVcBTxhjfgw0EhzzxxjzMvBja+2HwINAJrDAGLPz6y6w1n4axZxf8s5nNYwZlcvIYVluRXCNx+Oh6IKLCPh6yBw/3u04IiIyyKJWBFhrPwcO7+P4ab3+PjNaecJR09BOVX073zp+gttRoqp761bali9j2CmnklE+2u04IiISITE5MTBW/GNlLR4PHH5AkdtRoqaruorKu+dDIEDeUV8hNT/f7UgiIhIhKgJ2I+A4LP28jvEleeQnyZbBXRUVVP5iPni9lM27SQWAiEiCS77p7mFaV9lM7fYOjp2WHCsE7ti8iYq7bseTmkr5vB+RUZIc37eISDJTT8BuvP1JNUPSU5hhRrodJSp8dXWkZGVRev080guT43sWEUl2KgL60OMP8Mn6BqbtP4KM9BS340SUv7OTlMxMcmceRva0aXjT0t2OJCIiUaLhgD6s2tRIW6ePmQneC9BhP2fjTTfQvmolgAoAEZEkoyKgD+9+VkNWRipTxhe4HSViOlavouqXvyA1P1/j/yIiSUrDAbvw9fj5ZH0Dh08eSVpqYtZI7Z+toPqB+0kbWUTZD35Ial6e25FERMQFKgJ28fG6Brq6/cyYlJhDAV2VFVT/6j7SS0opu34eKTmDv+uiiIjEBxUBu1i+pp6czDQmjxnmdpSISC8to+Csb5A/62hSsrPdjiMiIi5KzP7uvdTW6WPZmnqmTyxMuB0DWz/6gO7aWjweD8O/eqoKABERURHQ2+rNjXT3BJg1tdjtKIOq+Z0l1Dz8IA0vPOt2FBERiSEaDuhl1abtZKSnMHZUrttRBk3zW3+n9nePkzXpAIouvMTtOCIiEkNUBIQ4jsOnGxqYPHoYqSmJ0UHS9Mbr1D31O7KmHETJv12DN13rAIiIyD8lxt1uEGyobmF7SxeHTBzhdpRB4fj9tLz/D7KnHULJ1d9XASAiIv9CPQEhH62pB2BqAiwQ5PT04ElNpeza6/GkpuFJ1T+ziIj8K/UEAP5AgPc+28rU/Qriftvghhefp/IXdxLo7sY7JFMFgIiI7JaKAGBtRTPN7d185aD4fSrAcRy2PfcnGp5/lrSCEbr5i4hIv3SnIDgUkJriYcq44W5H2SuO47DtmT/Q+MpfyZt9NEUXXIwnwdY5EBGRwZf0RUDAcXh/VS2H7F9IZkZ8Nsf2F5+n8ZW/kn/c8Yyc+x0VACIiEpb4vOsNour6dto6fUwZH5+9AAA5Mw7DcRwKvj4Hj8fjdhwREYkTSf+W8UNbhweYMi6+ngpwAgFaP1iK4zhklJQw4syzVACIiMiAJHUR4DgO/1hVixk9lGG58fNUgOP3s/W3j1Kz4EE6Vq10O46IiMSppC4Cqre1U9fYGVfbBjs9PWx9dAGt771LwZyzyT5wituRREQkTiX1nICVG7cDMCVOFghyenqoWfAQbcs/YsQ55zL8lNPcjiQiInEsqYuAD9fUUzoim5FDM92OEpYdGzfQtuJjCr/1bYadeLLbcUREJM4lbRFQ39TJuspm5swa53aUfjmOg8fjIXP/iYz92W2kF8bP8IWIiMSupJ0TsHR1LQBHTBnlcpI9C3R1UXXv3bR+9CGACgARERk0SVsErNy4ndFFOTE9FBDY0UnVvXfTsWolTne323FERCTBJGUR4Ovxs6G6hYnlQ92Oslv+jg4q77mbzvXrKL78KvKOPMrtSCIikmCSck7AJ+sa6O4JMHW/2HwqINDVReUv7qSrYgvFV11N7vRD3Y4kIiIJKCmLgI/XbSN7SCqTRg9zO0qfPOnpZE2aTMEZZ5Jz8DS344iISIJKyiLAbmli0phhpKbE1mhIT3MzgY520otLKDznXLfjiIhIgoutu2AUNLZ20dCyg/1K8t2O8iU9TY1U3nk7Vffdi9PT43YcERFJAklXBKyvagZgQlnsFAG+7Q1UzL8dX2MjRZd8F09qUnbQiIhIlCXd3Wb52noyM1IZV5zrdhQAfNvqqbxrPv72Nsquv4HM/Sa4HUlERJJEUvUEdHX7+eDzemZOKiTFGxvf+rbnn8Xf0U7Z9fNUAIiISFQlVU/A2somevyBmNo1sOj8C/GdcjoZpaVuRxERkSQTG2+Ho+TTDdtJ8XrYv9TdRYK6qqqofuB+Ajs68Q4ZogJARERckTQ9AY7j8Mm6bUweM4yM9BTXcnRVbKHy7jshJYWe5hbSh8TussUiIpLYkqYnoLaxk7qmTqbtP8K1DDs2baLizjvwpKVR/sObSC8qci2LiIhI0vQErN7cCODafgGdGzZQdc+deLOyKL/hJtIKC13JISIislPSFAErN26nIG8IpSOyXbl+SnY2GeWjGfXdK0griM09C0REJLkkxXCAPxBg9ebtHDB2GB6PJ6rX7q6pxnEc0ouKKJt3kwoAERGJGUlRBHy+pYnOLj8HjY/uDbh91Uo2//RWGv/2MkDUCxAREZE9SYrhgLUVTXiAKeOHR+2abSs+oebB+0kbVUzerNlRu66IiEi4kqMIqGymeEQ2Q9Kj8+22LV9G9cMPkFFaRtn180jJyYnKdUVERAYi4YcDevwB1lY2M2VcdHoBelpbqHl0AUPGjKHshh+qABARkZiV8D0BVfXt9PgDjCvOi8r1UnPzKL3mWjLGjCUlUwsBiYhI7Er4ImBNRRMAE0oju3Vw8ztL8KSmkHf4kWRNmhzRa4mIiAyGhB8O+HxLIwV5QyjIHxKxazS99Sa1jz9Gy3vv4ThOxK4jIiIymBK6COjxB1i1qTGiTwU0LV5E3f8+TvaUgyi5+t/1GKCIiMSNhB4OqN7WTpfPjxkdmaWCG1/9G/V/+D+ypx1C8ZX/hjctLSLXERERiYSELgK21LYBMKYoNyKv729vJ2fGTIovuxJPakI3pYiIJKCEvnNt3tpKRnoKRcOzBu01HcfB39JMav5QCuacDY6Dx5vQoyoiIpKgEvrutaGmmTEjc/AO0ji94zg0PPsnNv/XLfgaGvB4PCoAREQkbiXsHazL52fz1jb2H6Stgx3HYdsfn2b7yy+RM/1QUocNG5TXFRERcUvCDgdsqmkh4DiMH4RFghzHoX7hUzQtXsTQ40+gcO539BSAiIjEvYQtAjZUtwAMSk9A8xuv07R4EcNO+iojzv2WCgAREUkICVsEbK5tZXheBjmZ+/7YXt6so/GkpQX/qwJAREQSRELOCXAchw3VLYwdtfdDAY7fT8MLz+HvaMebnk7+7GNUAIiISEJJyCKgrrGTbc07OHDs3k3ec3p6qHnkYRpeeI625csHOZ2IiEhsSMjhgM21rQCMLxn4pkFOTw/VCx6kffkyRnzzPPK/Mmuw44mIiMSEhCwCbEUTGWkplBZmD+jrAr5uah56gPYVn1A493yGnXBShBKKiIi4LyGLgA3VLYwvySM1ZWCjHf62drqqqxh5wUUMPea4CKUTERGJDQlXBPT4A1TWtXHyYeVhf02gqwtPWhppw4Yx9if/gzcjI4IJRUREYkPCTQzctLUVf8BhQpjzAfydnVTdezd1v38SQAWAiIgkjYQrAqq3tQMwvqT/xwP9He1U3XMXnevXkWUmRTqaiIhITEm44YBtTZ3kZKaRn7Pnd/T+tjYq77mLrsoKSr53NTmHHBqlhCIiIrEh4YqAmsZOioZl7vEcx3Gouu8euqsqKbn6GnKmTotSOhERkdiRcEVAbUMH+/UzFODxeCj4+png8ZJ94JQoJRMREYktCVcE7OjuYVRBVp+f62lqpHPdWnJnHEb2lKlRTiYiIhJbEq4IACgb8a+LBPm2N1B513z8rS1kTTqAlJwcF5KJiIjEjoQsAnZdLthXX0/F3XcQaG+n9NofqAAQEREhAYuA/JwMMtJTvvi4u7aWyrvvILCji7If3MiQsWPdCyciIhJDEq4IKBo65Esft3/6CU63j/J5N5JRPtqlVCIiIrEn4YqA4oJgV7/j9+NJSWHYiSeTe9gRpOb1v3iQiIhIMkm4FQNHjcikq2ILm358Mzu2bAZQASAiItKHqPUEGGMmAk8ABUADcKG1du0u56QA9wGnAA5wu7X20YFcp6i7mYr77sE7JANvxpD+v0BERCRJRbMn4GHgAWvtROABYEEf55wPTAD2B44EbjXGjB3IRXr++ATerEzKf/gj0ouK9jGyiIhI4opKT4AxZiQwHTgpdGgh8CtjTKG1tr7XqecBj1hrA0C9MeY54JvAnWFcJgUgs6yUgm/OJXXo0MH7BuRLvF6P2xGSgto58tTGkac2jpxebZuyp/P2JFrDAeVAlbXWD2Ct9RtjqkPHexcBo4HNvT7eEjonHMUAB9w0b9/Tyh4VFGidhWhQO0ee2jjy1MZRUQys35svTKSnAz4AZgM1gN/lLCIiIpGWQrAA+GBvXyBaRUAFUGqMSQn1AqQAJaHjvW0BxvDPb2jXnoE96QKWDEZYERGROLFXPQA7RWVioLW2DvgYmBs6NBdYvst8AIA/ApcbY7zGmEJgDvBMNDKKiIgkm2g+HXAVcI0xZg1wTehjjDEvG2NmhM75HbABWAv8A/hva+3GKGYUERFJGh7HcdzOICIiIi5IuBUDRUREJDwqAkRERJKUigAREZEkpSJAREQkSakIEBERSVJxt2JgtHYjTGZhtvEtwLcIrs7oA2621r4S7azxYs+V5AAACE1JREFULJx27nWuAZYDD1prb4heyvgWbhsbY84FbgE8BH9nnGitrY1m1ngV5u+LkcBvCS4Dnwa8AXzfWtsT5bhxyRhzF/ANYCxwkLX2sz7O2av7Xjz2BERlN8IkF04bLwVmWmunApcCTxtjMqOYMRGE0847/+deADwXxWyJot82Dq1TcitwkrV2CjALaI5myDgXzs/xzcDq0O+LqcChwNnRixj3ngOOZs8r6O7VfS+uioBeuxEuDB1aCEwPrS7Y2xe7EYZWJdy5G6H0I9w2tta+Yq3tCH24guA7qIKoBY1zA/hZBrgJeAlYE6V4CWEAbXwdcJe1diuAtbbZWrsjeknj1wDa2AFyjTFeIANIB6qiFjTOWWuXWGt3XWZ/V3t134urIoA+diMEdu5G2Nu+7EaY7MJt494uBNZbayujkC9RhNXOxpiDga8C90Q9YfwL92f5AGC8MeYtY8wyY8z/M8Zo/9vwhNvGPwUmEtzgbSvwirX2nWgGTQJ7dd+LtyJAYowx5hiC/4PP7e9cGRhjTBrwa+Cqnb9kJSJSCHZRnwQcA5wKXOBqosTzTYI9hsVAKXC0MeYcdyMJxF8R8MVuhPDFWOmediPcaXQf50jfwm1jjDFHAk8Cc6y1Nqop41847VwM7Ae8bIzZBFxLcIOtX0c3atwayO+LZ6y1XdbaVuB54LCoJo1f4bbxNcBToa7qZoJtfFxUkya+vbrvxVURoN0IIy/cNjbGzASeBs6x1i6Lbsr4F047W2u3WGtHWGvHWmvHAvcSHPO7IuqB49AAfl/8HjjZGOMJ9b6cAHwSvaTxawBtvJHgrHWMMenAicC/zHCXfbJX9724KgJCtBth5IXTxg8CmcACY8zHoT8HuRM3boXTzrJvwmnj/wPqgFUEb2grgcdcyBqvwmnja4HZxphPCbbxGuARN8LGI2PMfcaYSqAMWGSMWRk6vs/3Pe0iKCIikqTisSdAREREBoGKABERkSSlIkBERCRJqQgQERFJUioCREREkpSKAJEYZox50hhzq9s5+mOMscaY2Xv4/KvGmPOjmUlE+hd3WwmLxKPQin9FBLde3mmitbbahSxPAucC3aE/HwL/bq3d6w2KrLWm1+v/DCiz1l7c6/Mn73Xg3TDGpBLcxrqD4AY1TQQ3sLnRWhsI4+tPBB4NLcQkkpTUEyASPWdYa3N6/Yl6AdDLz621OQQ3GNkO/MbFLPvqwND3cjzBNf8vcjmPSNxQT4CIi0Jbq/6B4B72QwiupvY9a+3qPs4dCTwOHAUEgM+stUeHPlcG3B96nTaCW+M+0N/1rbXtxpiFwBOh1xkCzCe44UuA4NLQN1lru/u5fiXwHSAH+CHgCW0QY621hxpjlgCPhl6vFjjMWvt56GtHEVxWtsxa22CM+TrBTanGEFxa9iprbb9LzFpr1xhj3gWm9Wqzy4AfEFxprQ64zVr7qDEmH3gRyDDGtIVOHw9sI7h183eBfGARwX+Pxv6uLxKP1BMg4r6XgP2BUQRver/bzXnzCC4LWhg69//BF4XES8AHBHdoOwmYZ4w5ob8LG2NygW8Dy0OHfgzMILir3iHAV4Af7en6vVlrXyJYRDwV6u04dJfPdxLc57z3rpPnAa+HCoCZBJeTvQwoINhD8Xxovfn+vpfJobzreh2uBU4H8oDLgfuNMVNDm9icAWzp1TNTB1wXOv9ogoVDG3Bff9cWiVfqCRCJnueMMT2hv79prZ0TGrt+fOcJoUmA9caYbGtt+y5f7yO4q+Boa+164K3Q8SOBPGvtz0MfrzPGPAZ8C3h9N1luMsZcC3QC7wOXho6fD1y+cwMYY8x/A78EfrKH6w/U7wneWP8r9PG3Q9cAuAJ40Fr7Qejj3xhj/hOYCexu//kVod3rsoCngAU7P2GtfbHXeYuNMa8Dswlua9uXq4DLrLVVAMaYnwBrjTEXhTPPQCTeqAgQiZ451tpFvQ+Ebl63AecAIwh2sxP6+65FwO0Eb8avG2P8wMPW2jsJdpuPNsY09To3BXhzD1lut9be2sfxEmBzr483E+xd2NP1B2oRMNQYcyjByXwHENxaFoLfy/nGmOt6nZ/eK0NfphLcRvU84GdANsEJjxhjvgbcQrCnxUuwUPig75cBgtuvvmiM2fWGPxLY2u93JhJnVASIuOtC4DSCk9o2E+wCrwc8u55orW0h2F19XWjHxjeMMUsJ7hm+1lo7eRDyVBO8EdvQx6OBqj1d31r7911eY4+7kllre4wxfyQ4JNAMvNCr16MC+Im19o6BhA69S19ojJkD/CdwgzEmk+BWqt8C/mKt9RljXuKfbdtXzkrg29ba9wdyfZF4pSJAxF25QBfQQPBd6v/s7kRjzBkEt7vdQPDm6SfYc/Ae0G2M+QHwAMFu+wOAdGvtRwPMsxD4sTFmGcGb5S3Ak/1cf1e1BLeN9Vhrd1cQ/J7gFr5twA29jj8C/MEYs5jgo4vZwHHA4j6GR/pyO/C2MeYOgu2QTrCo8od6BU4Ive7OnCOMMbnW2tbQsYeBnxtjLrHWbglNhjzCWvtCGNcWiTuaGCjirt8SfPddTXAf+3f3cK4BFhO8cb4D/NJa+7a1todgb8JhwCaCM9wXEJwMN1A/AT4hOEFxBcH5Arft6fp9vMbTBG++20M9FX15F+ghOMnw1Z0HrbX/AL4HPAQ0Etx3/jvhhrfWLidYFN1grW0i2HPxLMHHIM8hOIFy57mfAX8CNhljmkI3/F8AfyM45NEayjkz3OuLxBuP4+yx505EREQSlHoCREREkpSKABERkSSlIkBERCRJqQgQERFJUioCREREkpSKABERkSSlIkBERCRJqQgQERFJUv8fONUxgdBANjMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import metrics\r\n",
    "from sklearn.metrics import roc_auc_score\r\n",
    "\r\n",
    "\"\"\"预测并计算roc的相关指标\"\"\"\r\n",
    "val_pre_lgb = model.predict(X_val, num_iteration=model.best_iteration)\r\n",
    "fpr, tpr, threshold = metrics.roc_curve(y_val, val_pre_lgb)\r\n",
    "roc_auc = metrics.auc(fpr, tpr)\r\n",
    "print('未调参前lightgbm单模型在验证集上的AUC：{}'.format(roc_auc))\r\n",
    "\"\"\"画出roc曲线图\"\"\"\r\n",
    "plt.figure(figsize=(8, 8))\r\n",
    "plt.title('Validation ROC')\r\n",
    "plt.plot(fpr, tpr, 'b', label = 'Val AUC = %0.4f' % roc_auc)\r\n",
    "plt.ylim(0,1)\r\n",
    "plt.xlim(0,1)\r\n",
    "plt.legend(loc='best')\r\n",
    "plt.title('ROC')\r\n",
    "plt.ylabel('True Positive Rate')\r\n",
    "plt.xlabel('False Positive Rate')\r\n",
    "# 画出对角线\r\n",
    "plt.plot([0,1],[0,1],'r--')\r\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "************************************ 1 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "Early stopping, best iteration is:\n",
      "[121]\tvalid_0's auc: 0.718983\n",
      "[0.7189831988378461]\n",
      "************************************ 2 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "Early stopping, best iteration is:\n",
      "[105]\tvalid_0's auc: 0.715392\n",
      "[0.7189831988378461, 0.7153915866572624]\n",
      "************************************ 3 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "Early stopping, best iteration is:\n",
      "[93]\tvalid_0's auc: 0.719476\n",
      "[0.7189831988378461, 0.7153915866572624, 0.7194764444838756]\n",
      "************************************ 4 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "Early stopping, best iteration is:\n",
      "[109]\tvalid_0's auc: 0.718647\n",
      "[0.7189831988378461, 0.7153915866572624, 0.7194764444838756, 0.7186469467349867]\n",
      "************************************ 5 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "Early stopping, best iteration is:\n",
      "[117]\tvalid_0's auc: 0.719435\n",
      "[0.7189831988378461, 0.7153915866572624, 0.7194764444838756, 0.7186469467349867, 0.7194349846324942]\n",
      "lgb_scotrainre_list:[0.7189831988378461, 0.7153915866572624, 0.7194764444838756, 0.7186469467349867, 0.7194349846324942]\n",
      "lgb_score_mean:0.718386632269293\n",
      "lgb_score_std:0.0015284427925435235\n"
     ]
    }
   ],
   "source": [
    "import lightgbm as lgb\r\n",
    "\"\"\"使用lightgbm 5折交叉验证进行建模预测\"\"\"\r\n",
    "cv_scores = []\r\n",
    "for i, (train_index, valid_index) in enumerate(kf.split(X_train, y_train)):\r\n",
    "    print('************************************ {} ************************************'.format(str(i+1)))\r\n",
    "    X_train_split, y_train_split, X_val, y_val = X_train.iloc[train_index], y_train[train_index], X_train.iloc[valid_index], y_train[valid_index]\r\n",
    "    \r\n",
    "    train_matrix = lgb.Dataset(X_train_split, label=y_train_split)\r\n",
    "    valid_matrix = lgb.Dataset(X_val, label=y_val)\r\n",
    "\r\n",
    "    params = {\r\n",
    "                'boosting_type': 'gbdt',\r\n",
    "                'objective': 'binary',\r\n",
    "                'learning_rate': 0.1,\r\n",
    "                'metric': 'auc',\r\n",
    "        \r\n",
    "                'min_child_weight': 1e-3,\r\n",
    "                'num_leaves': 31,\r\n",
    "                'max_depth': -1,\r\n",
    "                'reg_lambda': 0,\r\n",
    "                'reg_alpha': 0,\r\n",
    "                'feature_fraction': 1,\r\n",
    "                'bagging_fraction': 1,\r\n",
    "                'bagging_freq': 0,\r\n",
    "                'seed': 2020,\r\n",
    "                'nthread': 8,\r\n",
    "                'silent': True,\r\n",
    "                'verbose': -1,\r\n",
    "    }\r\n",
    "    \r\n",
    "    model = lgb.train(params, train_set=train_matrix, num_boost_round=20000, valid_sets=valid_matrix, verbose_eval=1000, early_stopping_rounds=200)\r\n",
    "    val_pred = model.predict(X_val, num_iteration=model.best_iteration)\r\n",
    "    \r\n",
    "    cv_scores.append(roc_auc_score(y_val, val_pred))\r\n",
    "    print(cv_scores)\r\n",
    "\r\n",
    "print(\"lgb_scotrainre_list:{}\".format(cv_scores))\r\n",
    "print(\"lgb_score_mean:{}\".format(np.mean(cv_scores)))\r\n",
    "print(\"lgb_score_std:{}\".format(np.std(cv_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"通过网格搜索确定最优参数\"\"\"\r\n",
    "from sklearn.model_selection import GridSearchCV\r\n",
    "\r\n",
    "def get_best_cv_params(learning_rate=0.1, n_estimators=581, num_leaves=31, max_depth=-1, bagging_fraction=1.0, \r\n",
    "                       feature_fraction=1.0, bagging_freq=0, min_data_in_leaf=20, min_child_weight=0.001, \r\n",
    "                       min_split_gain=0, reg_lambda=0, reg_alpha=0, param_grid=None):\r\n",
    "    # 设置5折交叉验证\r\n",
    "    cv_fold = StratifiedKFold(n_splits=5, random_state=0, shuffle=True, )\r\n",
    "    \r\n",
    "    model_lgb = lgb.LGBMClassifier(learning_rate=learning_rate,\r\n",
    "                                   n_estimators=n_estimators,\r\n",
    "                                   num_leaves=num_leaves,\r\n",
    "                                   max_depth=max_depth,\r\n",
    "                                   bagging_fraction=bagging_fraction,\r\n",
    "                                   feature_fraction=feature_fraction,\r\n",
    "                                   bagging_freq=bagging_freq,\r\n",
    "                                   min_data_in_leaf=min_data_in_leaf,\r\n",
    "                                   min_child_weight=min_child_weight,\r\n",
    "                                   min_split_gain=min_split_gain,\r\n",
    "                                   reg_lambda=reg_lambda,\r\n",
    "                                   reg_alpha=reg_alpha,\r\n",
    "                                   n_jobs= 8\r\n",
    "                                  )\r\n",
    "    grid_search = GridSearchCV(estimator=model_lgb, \r\n",
    "                               cv=cv_fold,\r\n",
    "                               param_grid=param_grid,\r\n",
    "                               scoring='roc_auc'\r\n",
    "                              )\r\n",
    "    grid_search.fit(X_train, y_train)\r\n",
    "\r\n",
    "    print('模型当前最优参数为:{}'.format(grid_search.best_params_))\r\n",
    "    print('模型当前最优得分为:{}'.format(grid_search.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"以下代码未运行，耗时较长，请谨慎运行，且每一步的最优参数需要在下一步进行手动更新，请注意\"\"\"\r\n",
    "\r\n",
    "\"\"\"\r\n",
    "需要注意一下的是，除了获取上面的获取num_boost_round时候用的是原生的lightgbm（因为要用自带的cv）\r\n",
    "下面配合GridSearchCV时必须使用sklearn接口的lightgbm。\r\n",
    "\"\"\"\r\n",
    "\"\"\"设置n_estimators 为581，调整num_leaves和max_depth，这里选择先粗调再细调\"\"\"\r\n",
    "lgb_params = {'num_leaves': range(10, 80, 5), 'max_depth': range(3,10,2)}\r\n",
    "get_best_cv_params(learning_rate=0.1, n_estimators=581, num_leaves=None, max_depth=None, min_data_in_leaf=20, \r\n",
    "                   min_child_weight=0.001,bagging_fraction=1.0, feature_fraction=1.0, bagging_freq=0, \r\n",
    "                   min_split_gain=0, reg_lambda=0, reg_alpha=0, param_grid=lgb_params)\r\n",
    "\r\n",
    "\"\"\"num_leaves为30，max_depth为7，进一步细调num_leaves和max_depth\"\"\"\r\n",
    "lgb_params = {'num_leaves': range(25, 35, 1), 'max_depth': range(5,9,1)}\r\n",
    "get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=None, max_depth=None, min_data_in_leaf=20, \r\n",
    "                   min_child_weight=0.001,bagging_fraction=1.0, feature_fraction=1.0, bagging_freq=0, \r\n",
    "                   min_split_gain=0, reg_lambda=0, reg_alpha=0, param_grid=lgb_params)\r\n",
    "\r\n",
    "\"\"\"\r\n",
    "确定min_data_in_leaf为45，min_child_weight为0.001 ，下面进行bagging_fraction、feature_fraction和bagging_freq的调参\r\n",
    "\"\"\"\r\n",
    "lgb_params = {'bagging_fraction': [i/10 for i in range(5,10,1)], \r\n",
    "              'feature_fraction': [i/10 for i in range(5,10,1)],\r\n",
    "              'bagging_freq': range(0,81,10)\r\n",
    "             }\r\n",
    "get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=29, max_depth=7, min_data_in_leaf=45, \r\n",
    "                   min_child_weight=0.001,bagging_fraction=None, feature_fraction=None, bagging_freq=None, \r\n",
    "                   min_split_gain=0, reg_lambda=0, reg_alpha=0, param_grid=lgb_params)\r\n",
    "\r\n",
    "\"\"\"\r\n",
    "确定bagging_fraction为0.4、feature_fraction为0.6、bagging_freq为 ，下面进行reg_lambda、reg_alpha的调参\r\n",
    "\"\"\"\r\n",
    "lgb_params = {'reg_lambda': [0,0.001,0.01,0.03,0.08,0.3,0.5], 'reg_alpha': [0,0.001,0.01,0.03,0.08,0.3,0.5]}\r\n",
    "get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=29, max_depth=7, min_data_in_leaf=45, \r\n",
    "                   min_child_weight=0.001,bagging_fraction=0.9, feature_fraction=0.9, bagging_freq=40, \r\n",
    "                   min_split_gain=0, reg_lambda=None, reg_alpha=None, param_grid=lgb_params)\r\n",
    "\r\n",
    "\"\"\"\r\n",
    "确定reg_lambda、reg_alpha都为0，下面进行min_split_gain的调参\r\n",
    "\"\"\"\r\n",
    "lgb_params = {'min_split_gain': [i/10 for i in range(0,11,1)]}\r\n",
    "get_best_cv_params(learning_rate=0.1, n_estimators=85, num_leaves=29, max_depth=7, min_data_in_leaf=45, \r\n",
    "                   min_child_weight=0.001,bagging_fraction=0.9, feature_fraction=0.9, bagging_freq=40, \r\n",
    "                   min_split_gain=None, reg_lambda=0, reg_alpha=0, param_grid=lgb_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\r\n",
    "参数确定好了以后，我们设置一个比较小的learning_rate 0.005，来确定最终的num_boost_round\r\n",
    "\"\"\"\r\n",
    "# 设置5折交叉验证\r\n",
    "# cv_fold = StratifiedKFold(n_splits=5, random_state=0, shuffle=True, )\r\n",
    "final_params = {\r\n",
    "                'boosting_type': 'gbdt',\r\n",
    "                'learning_rate': 0.01,\r\n",
    "                'num_leaves': 29,\r\n",
    "                'max_depth': 7,\r\n",
    "                'min_data_in_leaf':45,\r\n",
    "                'min_child_weight':0.001,\r\n",
    "                'bagging_fraction': 0.9,\r\n",
    "                'feature_fraction': 0.9,\r\n",
    "                'bagging_freq': 40,\r\n",
    "                'min_split_gain': 0,\r\n",
    "                'reg_lambda':0,\r\n",
    "                'reg_alpha':0,\r\n",
    "                'nthread': 6\r\n",
    "               }\r\n",
    "\r\n",
    "cv_result = lgb.cv(train_set=lgb_train,\r\n",
    "                   early_stopping_rounds=20,\r\n",
    "                   num_boost_round=5000,\r\n",
    "                   nfold=5,\r\n",
    "                   stratified=True,\r\n",
    "                   shuffle=True,\r\n",
    "                   params=final_params,\r\n",
    "                   metrics='auc',\r\n",
    "                   seed=0,\r\n",
    "                  )\r\n",
    "\r\n",
    "print('迭代次数{}'.format(len(cv_result['auc-mean'])))\r\n",
    "print('交叉验证的AUC为{}'.format(max(cv_result['auc-mean'])))\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 贝叶斯调参\n",
    "\n",
    "安装所需要的库：\n",
    "\n",
    "pip install bayesian-optimization\n",
    "\n",
    "贝叶斯调参步骤：\n",
    "\n",
    "1.定义优化函数(rf_cv）\n",
    "\n",
    "2.建立模型\n",
    "\n",
    "3.定义待优化的参数\n",
    "\n",
    "4.得到优化结果，并返回要优化的分数指标\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\r\n",
    "\r\n",
    "\"\"\"定义优化函数\"\"\"\r\n",
    "def rf_cv_lgb(num_leaves, max_depth, bagging_fraction, feature_fraction, bagging_freq, min_data_in_leaf, \r\n",
    "              min_child_weight, min_split_gain, reg_lambda, reg_alpha):\r\n",
    "    # 建立模型\r\n",
    "    model_lgb = lgb.LGBMClassifier(boosting_type='gbdt', bjective='binary', metric='auc',\r\n",
    "                                   learning_rate=0.1, n_estimators=5000,\r\n",
    "                                   num_leaves=int(num_leaves), max_depth=int(max_depth), \r\n",
    "                                   bagging_fraction=round(bagging_fraction, 2), feature_fraction=round(feature_fraction, 2),\r\n",
    "                                   bagging_freq=int(bagging_freq), min_data_in_leaf=int(min_data_in_leaf),\r\n",
    "                                   min_child_weight=min_child_weight, min_split_gain=min_split_gain,\r\n",
    "                                   reg_lambda=reg_lambda, reg_alpha=reg_alpha,\r\n",
    "                                   n_jobs= 8\r\n",
    "                                  )\r\n",
    "    \r\n",
    "    val = cross_val_score(model_lgb, X_train_split, y_train_split, cv=5, scoring='roc_auc').mean()\r\n",
    "    \r\n",
    "    return val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   | baggin... | baggin... | featur... | max_depth | min_ch... | min_da... | min_sp... | num_le... | reg_alpha | reg_la... |\n",
      "-------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.63, subsample=1.0 will be ignored. Current value: bagging_fraction=0.63\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.63, subsample=1.0 will be ignored. Current value: bagging_fraction=0.63\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.63, subsample=1.0 will be ignored. Current value: bagging_fraction=0.63\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.63, subsample=1.0 will be ignored. Current value: bagging_fraction=0.63\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.63, subsample=1.0 will be ignored. Current value: bagging_fraction=0.63\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m 0.7086  \u001b[0m | \u001b[0m 0.634   \u001b[0m | \u001b[0m 15.57   \u001b[0m | \u001b[0m 0.683   \u001b[0m | \u001b[0m 13.73   \u001b[0m | \u001b[0m 3.515   \u001b[0m | \u001b[0m 58.44   \u001b[0m | \u001b[0m 0.9845  \u001b[0m | \u001b[0m 14.2    \u001b[0m | \u001b[0m 1.827   \u001b[0m | \u001b[0m 3.933   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.95, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.95\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=69, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=69\n",
      "[LightGBM] [Warning] bagging_freq is set=22, subsample_freq=0 will be ignored. Current value: bagging_freq=22\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.95, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.95\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=69, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=69\n",
      "[LightGBM] [Warning] bagging_freq is set=22, subsample_freq=0 will be ignored. Current value: bagging_freq=22\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.95, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.95\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=69, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=69\n",
      "[LightGBM] [Warning] bagging_freq is set=22, subsample_freq=0 will be ignored. Current value: bagging_freq=22\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.95, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.95\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=69, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=69\n",
      "[LightGBM] [Warning] bagging_freq is set=22, subsample_freq=0 will be ignored. Current value: bagging_freq=22\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.95, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.95\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=69, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=69\n",
      "[LightGBM] [Warning] bagging_freq is set=22, subsample_freq=0 will be ignored. Current value: bagging_freq=22\n",
      "| \u001b[0m 2       \u001b[0m | \u001b[0m 0.6893  \u001b[0m | \u001b[0m 0.6972  \u001b[0m | \u001b[0m 22.48   \u001b[0m | \u001b[0m 0.9539  \u001b[0m | \u001b[0m 18.33   \u001b[0m | \u001b[0m 2.234   \u001b[0m | \u001b[0m 69.13   \u001b[0m | \u001b[0m 0.9769  \u001b[0m | \u001b[0m 27.35   \u001b[0m | \u001b[0m 7.525   \u001b[0m | \u001b[0m 1.074   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=77, subsample_freq=0 will be ignored. Current value: bagging_freq=77\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=77, subsample_freq=0 will be ignored. Current value: bagging_freq=77\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=77, subsample_freq=0 will be ignored. Current value: bagging_freq=77\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=77, subsample_freq=0 will be ignored. Current value: bagging_freq=77\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.59, subsample=1.0 will be ignored. Current value: bagging_fraction=0.59\n",
      "[LightGBM] [Warning] feature_fraction is set=0.89, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.89\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=77, subsample_freq=0 will be ignored. Current value: bagging_freq=77\n",
      "| \u001b[95m 3       \u001b[0m | \u001b[95m 0.7099  \u001b[0m | \u001b[95m 0.5926  \u001b[0m | \u001b[95m 77.63   \u001b[0m | \u001b[95m 0.8939  \u001b[0m | \u001b[95m 10.82   \u001b[0m | \u001b[95m 2.886   \u001b[0m | \u001b[95m 81.68   \u001b[0m | \u001b[95m 0.9307  \u001b[0m | \u001b[95m 165.7   \u001b[0m | \u001b[95m 1.196   \u001b[0m | \u001b[95m 7.551   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=39, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=39\n",
      "[LightGBM] [Warning] bagging_freq is set=35, subsample_freq=0 will be ignored. Current value: bagging_freq=35\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=39, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=39\n",
      "[LightGBM] [Warning] bagging_freq is set=35, subsample_freq=0 will be ignored. Current value: bagging_freq=35\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=39, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=39\n",
      "[LightGBM] [Warning] bagging_freq is set=35, subsample_freq=0 will be ignored. Current value: bagging_freq=35\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=39, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=39\n",
      "[LightGBM] [Warning] bagging_freq is set=35, subsample_freq=0 will be ignored. Current value: bagging_freq=35\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=1.0, subsample=1.0 will be ignored. Current value: bagging_fraction=1.0\n",
      "[LightGBM] [Warning] feature_fraction is set=0.7, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.7\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=39, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=39\n",
      "[LightGBM] [Warning] bagging_freq is set=35, subsample_freq=0 will be ignored. Current value: bagging_freq=35\n",
      "| \u001b[95m 4       \u001b[0m | \u001b[95m 0.7217  \u001b[0m | \u001b[95m 0.9991  \u001b[0m | \u001b[95m 35.18   \u001b[0m | \u001b[95m 0.7017  \u001b[0m | \u001b[95m 12.82   \u001b[0m | \u001b[95m 8.749   \u001b[0m | \u001b[95m 39.76   \u001b[0m | \u001b[95m 0.7021  \u001b[0m | \u001b[95m 55.78   \u001b[0m | \u001b[95m 8.8     \u001b[0m | \u001b[95m 0.7648  \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.75, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.75\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=70, subsample_freq=0 will be ignored. Current value: bagging_freq=70\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.75, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.75\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=70, subsample_freq=0 will be ignored. Current value: bagging_freq=70\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.75, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.75\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=70, subsample_freq=0 will be ignored. Current value: bagging_freq=70\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.75, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.75\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=70, subsample_freq=0 will be ignored. Current value: bagging_freq=70\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.75, subsample=1.0 will be ignored. Current value: bagging_fraction=0.75\n",
      "[LightGBM] [Warning] feature_fraction is set=0.75, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.75\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=70, subsample_freq=0 will be ignored. Current value: bagging_freq=70\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m 0.6872  \u001b[0m | \u001b[0m 0.7458  \u001b[0m | \u001b[0m 70.41   \u001b[0m | \u001b[0m 0.7469  \u001b[0m | \u001b[0m 9.934   \u001b[0m | \u001b[0m 6.483   \u001b[0m | \u001b[0m 98.52   \u001b[0m | \u001b[0m 0.1471  \u001b[0m | \u001b[0m 184.8   \u001b[0m | \u001b[0m 0.7032  \u001b[0m | \u001b[0m 2.128   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.98, subsample=1.0 will be ignored. Current value: bagging_fraction=0.98\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.98, subsample=1.0 will be ignored. Current value: bagging_fraction=0.98\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.98, subsample=1.0 will be ignored. Current value: bagging_fraction=0.98\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.98, subsample=1.0 will be ignored. Current value: bagging_fraction=0.98\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.98, subsample=1.0 will be ignored. Current value: bagging_fraction=0.98\n",
      "[LightGBM] [Warning] feature_fraction is set=0.68, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.68\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=10, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=10\n",
      "[LightGBM] [Warning] bagging_freq is set=96, subsample_freq=0 will be ignored. Current value: bagging_freq=96\n",
      "| \u001b[0m 6       \u001b[0m | \u001b[0m 0.72    \u001b[0m | \u001b[0m 0.9757  \u001b[0m | \u001b[0m 96.8    \u001b[0m | \u001b[0m 0.681   \u001b[0m | \u001b[0m 4.101   \u001b[0m | \u001b[0m 3.064   \u001b[0m | \u001b[0m 10.0    \u001b[0m | \u001b[0m 0.7847  \u001b[0m | \u001b[0m 14.32   \u001b[0m | \u001b[0m 0.3308  \u001b[0m | \u001b[0m 5.887   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.69, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.69\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=14, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=14\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.69, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.69\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=14, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=14\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.69, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.69\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=14, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=14\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.69, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.69\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=14, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=14\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.69, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.69\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=14, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=14\n",
      "[LightGBM] [Warning] bagging_freq is set=5, subsample_freq=0 will be ignored. Current value: bagging_freq=5\n",
      "| \u001b[0m 7       \u001b[0m | \u001b[0m 0.6947  \u001b[0m | \u001b[0m 0.8342  \u001b[0m | \u001b[0m 5.714   \u001b[0m | \u001b[0m 0.6888  \u001b[0m | \u001b[0m 14.19   \u001b[0m | \u001b[0m 1.475   \u001b[0m | \u001b[0m 14.66   \u001b[0m | \u001b[0m 0.8061  \u001b[0m | \u001b[0m 196.8   \u001b[0m | \u001b[0m 5.368   \u001b[0m | \u001b[0m 3.264   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.58, subsample=1.0 will be ignored. Current value: bagging_fraction=0.58\n",
      "[LightGBM] [Warning] feature_fraction is set=0.82, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.82\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=15, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=15\n",
      "[LightGBM] [Warning] bagging_freq is set=98, subsample_freq=0 will be ignored. Current value: bagging_freq=98\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.58, subsample=1.0 will be ignored. Current value: bagging_fraction=0.58\n",
      "[LightGBM] [Warning] feature_fraction is set=0.82, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.82\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=15, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=15\n",
      "[LightGBM] [Warning] bagging_freq is set=98, subsample_freq=0 will be ignored. Current value: bagging_freq=98\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.58, subsample=1.0 will be ignored. Current value: bagging_fraction=0.58\n",
      "[LightGBM] [Warning] feature_fraction is set=0.82, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.82\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=15, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=15\n",
      "[LightGBM] [Warning] bagging_freq is set=98, subsample_freq=0 will be ignored. Current value: bagging_freq=98\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.58, subsample=1.0 will be ignored. Current value: bagging_fraction=0.58\n",
      "[LightGBM] [Warning] feature_fraction is set=0.82, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.82\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=15, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=15\n",
      "[LightGBM] [Warning] bagging_freq is set=98, subsample_freq=0 will be ignored. Current value: bagging_freq=98\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.58, subsample=1.0 will be ignored. Current value: bagging_fraction=0.58\n",
      "[LightGBM] [Warning] feature_fraction is set=0.82, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.82\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=15, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=15\n",
      "[LightGBM] [Warning] bagging_freq is set=98, subsample_freq=0 will be ignored. Current value: bagging_freq=98\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m 0.7025  \u001b[0m | \u001b[0m 0.5787  \u001b[0m | \u001b[0m 98.34   \u001b[0m | \u001b[0m 0.8158  \u001b[0m | \u001b[0m 18.88   \u001b[0m | \u001b[0m 7.848   \u001b[0m | \u001b[0m 15.38   \u001b[0m | \u001b[0m 0.6384  \u001b[0m | \u001b[0m 198.8   \u001b[0m | \u001b[0m 0.3308  \u001b[0m | \u001b[0m 7.896   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.78, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.78\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=13, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=13\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.78, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.78\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=13, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=13\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.78, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.78\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=13, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=13\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.78, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.78\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=13, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=13\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.83, subsample=1.0 will be ignored. Current value: bagging_fraction=0.83\n",
      "[LightGBM] [Warning] feature_fraction is set=0.78, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.78\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=13, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=13\n",
      "[LightGBM] [Warning] bagging_freq is set=15, subsample_freq=0 will be ignored. Current value: bagging_freq=15\n",
      "| \u001b[0m 9       \u001b[0m | \u001b[0m 0.7127  \u001b[0m | \u001b[0m 0.8279  \u001b[0m | \u001b[0m 15.52   \u001b[0m | \u001b[0m 0.7789  \u001b[0m | \u001b[0m 4.131   \u001b[0m | \u001b[0m 5.326   \u001b[0m | \u001b[0m 13.3    \u001b[0m | \u001b[0m 0.9348  \u001b[0m | \u001b[0m 12.26   \u001b[0m | \u001b[0m 8.423   \u001b[0m | \u001b[0m 9.968   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.92, subsample=1.0 will be ignored. Current value: bagging_fraction=0.92\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=35, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=35\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.92, subsample=1.0 will be ignored. Current value: bagging_fraction=0.92\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=35, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=35\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.92, subsample=1.0 will be ignored. Current value: bagging_fraction=0.92\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=35, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=35\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.92, subsample=1.0 will be ignored. Current value: bagging_fraction=0.92\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=35, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=35\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.92, subsample=1.0 will be ignored. Current value: bagging_fraction=0.92\n",
      "[LightGBM] [Warning] feature_fraction is set=0.81, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.81\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=35, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=35\n",
      "[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8\n",
      "| \u001b[0m 10      \u001b[0m | \u001b[0m 0.711   \u001b[0m | \u001b[0m 0.921   \u001b[0m | \u001b[0m 8.473   \u001b[0m | \u001b[0m 0.8098  \u001b[0m | \u001b[0m 4.349   \u001b[0m | \u001b[0m 8.891   \u001b[0m | \u001b[0m 35.83   \u001b[0m | \u001b[0m 0.6688  \u001b[0m | \u001b[0m 97.71   \u001b[0m | \u001b[0m 0.1491  \u001b[0m | \u001b[0m 9.737   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.79, subsample=1.0 will be ignored. Current value: bagging_fraction=0.79\n",
      "[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=85, subsample_freq=0 will be ignored. Current value: bagging_freq=85\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.79, subsample=1.0 will be ignored. Current value: bagging_fraction=0.79\n",
      "[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=85, subsample_freq=0 will be ignored. Current value: bagging_freq=85\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.79, subsample=1.0 will be ignored. Current value: bagging_fraction=0.79\n",
      "[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=85, subsample_freq=0 will be ignored. Current value: bagging_freq=85\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.79, subsample=1.0 will be ignored. Current value: bagging_fraction=0.79\n",
      "[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=85, subsample_freq=0 will be ignored. Current value: bagging_freq=85\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.79, subsample=1.0 will be ignored. Current value: bagging_fraction=0.79\n",
      "[LightGBM] [Warning] feature_fraction is set=0.87, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.87\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=98, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=98\n",
      "[LightGBM] [Warning] bagging_freq is set=85, subsample_freq=0 will be ignored. Current value: bagging_freq=85\n",
      "| \u001b[0m 11      \u001b[0m | \u001b[0m 0.7132  \u001b[0m | \u001b[0m 0.7927  \u001b[0m | \u001b[0m 86.0    \u001b[0m | \u001b[0m 0.8713  \u001b[0m | \u001b[0m 5.811   \u001b[0m | \u001b[0m 9.427   \u001b[0m | \u001b[0m 98.37   \u001b[0m | \u001b[0m 0.7606  \u001b[0m | \u001b[0m 15.82   \u001b[0m | \u001b[0m 8.355   \u001b[0m | \u001b[0m 5.514   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.67, subsample=1.0 will be ignored. Current value: bagging_fraction=0.67\n",
      "[LightGBM] [Warning] feature_fraction is set=0.55, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.55\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=12, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=12\n",
      "[LightGBM] [Warning] bagging_freq is set=97, subsample_freq=0 will be ignored. Current value: bagging_freq=97\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.67, subsample=1.0 will be ignored. Current value: bagging_fraction=0.67\n",
      "[LightGBM] [Warning] feature_fraction is set=0.55, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.55\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=12, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=12\n",
      "[LightGBM] [Warning] bagging_freq is set=97, subsample_freq=0 will be ignored. Current value: bagging_freq=97\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.67, subsample=1.0 will be ignored. Current value: bagging_fraction=0.67\n",
      "[LightGBM] [Warning] feature_fraction is set=0.55, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.55\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=12, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=12\n",
      "[LightGBM] [Warning] bagging_freq is set=97, subsample_freq=0 will be ignored. Current value: bagging_freq=97\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.67, subsample=1.0 will be ignored. Current value: bagging_fraction=0.67\n",
      "[LightGBM] [Warning] feature_fraction is set=0.55, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.55\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=12, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=12\n",
      "[LightGBM] [Warning] bagging_freq is set=97, subsample_freq=0 will be ignored. Current value: bagging_freq=97\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.67, subsample=1.0 will be ignored. Current value: bagging_fraction=0.67\n",
      "[LightGBM] [Warning] feature_fraction is set=0.55, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.55\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=12, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=12\n",
      "[LightGBM] [Warning] bagging_freq is set=97, subsample_freq=0 will be ignored. Current value: bagging_freq=97\n",
      "| \u001b[0m 12      \u001b[0m | \u001b[0m 0.7186  \u001b[0m | \u001b[0m 0.6715  \u001b[0m | \u001b[0m 97.05   \u001b[0m | \u001b[0m 0.5511  \u001b[0m | \u001b[0m 3.914   \u001b[0m | \u001b[0m 1.175   \u001b[0m | \u001b[0m 12.41   \u001b[0m | \u001b[0m 0.26    \u001b[0m | \u001b[0m 129.2   \u001b[0m | \u001b[0m 7.667   \u001b[0m | \u001b[0m 1.834   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.77, subsample=1.0 will be ignored. Current value: bagging_fraction=0.77\n",
      "[LightGBM] [Warning] feature_fraction is set=0.66, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.66\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=37, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=37\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.77, subsample=1.0 will be ignored. Current value: bagging_fraction=0.77\n",
      "[LightGBM] [Warning] feature_fraction is set=0.66, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.66\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=37, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=37\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.77, subsample=1.0 will be ignored. Current value: bagging_fraction=0.77\n",
      "[LightGBM] [Warning] feature_fraction is set=0.66, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.66\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=37, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=37\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.77, subsample=1.0 will be ignored. Current value: bagging_fraction=0.77\n",
      "[LightGBM] [Warning] feature_fraction is set=0.66, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.66\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=37, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=37\n",
      "[LightGBM] [Warning] bagging_freq is set=37, subsample_freq=0 will be ignored. Current value: bagging_freq=37\n",
      "| \u001b[0m 13      \u001b[0m | \u001b[0m 0.6895  \u001b[0m | \u001b[0m 0.7741  \u001b[0m | \u001b[0m 37.69   \u001b[0m | \u001b[0m 0.6613  \u001b[0m | \u001b[0m 11.68   \u001b[0m | \u001b[0m 7.915   \u001b[0m | \u001b[0m 37.4    \u001b[0m | \u001b[0m 0.1317  \u001b[0m | \u001b[0m 55.98   \u001b[0m | \u001b[0m 7.331   \u001b[0m | \u001b[0m 1.104   \u001b[0m |\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.68, subsample=1.0 will be ignored. Current value: bagging_fraction=0.68\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=11, subsample_freq=0 will be ignored. Current value: bagging_freq=11\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.68, subsample=1.0 will be ignored. Current value: bagging_fraction=0.68\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=11, subsample_freq=0 will be ignored. Current value: bagging_freq=11\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.68, subsample=1.0 will be ignored. Current value: bagging_fraction=0.68\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=11, subsample_freq=0 will be ignored. Current value: bagging_freq=11\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.68, subsample=1.0 will be ignored. Current value: bagging_fraction=0.68\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=11, subsample_freq=0 will be ignored. Current value: bagging_freq=11\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.68, subsample=1.0 will be ignored. Current value: bagging_fraction=0.68\n",
      "[LightGBM] [Warning] feature_fraction is set=0.54, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.54\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=58, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=58\n",
      "[LightGBM] [Warning] bagging_freq is set=11, subsample_freq=0 will be ignored. Current value: bagging_freq=11\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.88, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.88\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=73, subsample_freq=0 will be ignored. Current value: bagging_freq=73\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.88, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.88\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=73, subsample_freq=0 will be ignored. Current value: bagging_freq=73\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.88, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.88\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=73, subsample_freq=0 will be ignored. Current value: bagging_freq=73\n",
      "[LightGBM] [Warning] Unknown parameter: bjective\n",
      "[LightGBM] [Warning] bagging_fraction is set=0.7, subsample=1.0 will be ignored. Current value: bagging_fraction=0.7\n",
      "[LightGBM] [Warning] feature_fraction is set=0.88, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.88\n",
      "[LightGBM] [Warning] min_data_in_leaf is set=81, min_child_samples=20 will be ignored. Current value: min_data_in_leaf=81\n",
      "[LightGBM] [Warning] bagging_freq is set=73, subsample_freq=0 will be ignored. Current value: bagging_freq=73\n",
      "| \u001b[0m 15      \u001b[0m | \u001b[0m 0.7046  \u001b[0m | \u001b[0m 0.7002  \u001b[0m | \u001b[0m 73.89   \u001b[0m | \u001b[0m 0.8763  \u001b[0m | \u001b[0m 13.84   \u001b[0m | \u001b[0m 2.031   \u001b[0m | \u001b[0m 81.42   \u001b[0m | \u001b[0m 0.4852  \u001b[0m | \u001b[0m 166.4   \u001b[0m | \u001b[0m 2.923   \u001b[0m | \u001b[0m 5.923   \u001b[0m |\n",
      "=================================================================================================================================================\n"
     ]
    }
   ],
   "source": [
    "from bayes_opt import BayesianOptimization\r\n",
    "\"\"\"定义优化参数\"\"\"\r\n",
    "bayes_lgb = BayesianOptimization(\r\n",
    "    rf_cv_lgb, \r\n",
    "    {\r\n",
    "        'num_leaves':(10, 200),\r\n",
    "        'max_depth':(3, 20),\r\n",
    "        'bagging_fraction':(0.5, 1.0),\r\n",
    "        'feature_fraction':(0.5, 1.0),\r\n",
    "        'bagging_freq':(0, 100),\r\n",
    "        'min_data_in_leaf':(10,100),\r\n",
    "        'min_child_weight':(0, 10),\r\n",
    "        'min_split_gain':(0.0, 1.0),\r\n",
    "        'reg_alpha':(0.0, 10),\r\n",
    "        'reg_lambda':(0.0, 10),\r\n",
    "    }\r\n",
    ")\r\n",
    "\r\n",
    "\"\"\"开始优化\"\"\"\r\n",
    "\r\n",
    "bayes_lgb.maximize(n_iter=10)\r\n",
    "#运行时长: 14小时9分钟59秒"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'target': 0.7216559541217007,\n",
       " 'params': {'bagging_fraction': 0.9990618012082493,\n",
       "  'bagging_freq': 35.1837365729197,\n",
       "  'feature_fraction': 0.701668450472329,\n",
       "  'max_depth': 12.81556694510836,\n",
       "  'min_child_weight': 8.749156084914016,\n",
       "  'min_data_in_leaf': 39.75537683778964,\n",
       "  'min_split_gain': 0.7020906485191929,\n",
       "  'num_leaves': 55.77679262019952,\n",
       "  'reg_alpha': 8.799845190246767,\n",
       "  'reg_lambda': 0.764784406363006}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"显示优化结果\"\"\"\r\n",
    "bayes_lgb.max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "迭代次数3229\n",
      "最终模型的AUC为0.7214851054528417\n"
     ]
    }
   ],
   "source": [
    "\"\"\"调整一个较小的学习率，并通过cv函数确定当前最优的迭代次数\"\"\"\r\n",
    "base_params_lgb = {\r\n",
    "                    'boosting_type': 'gbdt',\r\n",
    "                    'objective': 'binary',\r\n",
    "                    'metric': 'auc',\r\n",
    "                    'learning_rate': 0.01,\r\n",
    "                    'num_leaves': 14,\r\n",
    "                    'max_depth': 19,\r\n",
    "                    'min_data_in_leaf': 37,\r\n",
    "                    'min_child_weight':1.6,\r\n",
    "                    'bagging_fraction': 0.98,\r\n",
    "                    'feature_fraction': 0.69,\r\n",
    "                    'bagging_freq': 96,\r\n",
    "                    'reg_lambda': 9,\r\n",
    "                    'reg_alpha': 7,\r\n",
    "                    'min_split_gain': 0.4,\r\n",
    "                    'nthread': 8,\r\n",
    "                    'seed': 2020,\r\n",
    "                    'silent': True,\r\n",
    "                    'verbose': -1,\r\n",
    "}\r\n",
    "\r\n",
    "cv_result_lgb = lgb.cv(\r\n",
    "    train_set=train_matrix,\r\n",
    "    early_stopping_rounds=1000, \r\n",
    "    num_boost_round=20000,\r\n",
    "    nfold=5,\r\n",
    "    stratified=True,\r\n",
    "    shuffle=True,\r\n",
    "    params=base_params_lgb,\r\n",
    "    metrics='auc',\r\n",
    "    seed=0\r\n",
    ")\r\n",
    "\r\n",
    "print('迭代次数{}'.format(len(cv_result_lgb['auc-mean'])))\r\n",
    "print('最终模型的AUC为{}'.format(max(cv_result_lgb['auc-mean'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "************************************ 1 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] Number of positive: 128011, number of negative: 511989\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.123911 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4025\n",
      "[LightGBM] [Info] Number of data points in the train set: 640000, number of used features: 43\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.200017 -> initscore=-1.386187\n",
      "[LightGBM] [Info] Start training from score -1.386187\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[1000]\tvalid_0's auc: 0.721094\n",
      "[2000]\tvalid_0's auc: 0.723595\n",
      "[3000]\tvalid_0's auc: 0.724267\n",
      "Early stopping, best iteration is:\n",
      "[3436]\tvalid_0's auc: 0.724399\n",
      "[0.7243989768290735]\n",
      "************************************ 2 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] Number of positive: 127533, number of negative: 512467\n",
      "[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.145222 seconds.\n",
      "You can set `force_row_wise=true` to remove the overhead.\n",
      "And if memory is not enough, you can set `force_col_wise=true`.\n",
      "[LightGBM] [Info] Total Bins 4019\n",
      "[LightGBM] [Info] Number of data points in the train set: 640000, number of used features: 43\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.199270 -> initscore=-1.390861\n",
      "[LightGBM] [Info] Start training from score -1.390861\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[1000]\tvalid_0's auc: 0.717451\n",
      "[2000]\tvalid_0's auc: 0.719761\n",
      "[3000]\tvalid_0's auc: 0.72023\n",
      "Early stopping, best iteration is:\n",
      "[3273]\tvalid_0's auc: 0.720405\n",
      "[0.7243989768290735, 0.720405047892458]\n",
      "************************************ 3 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] Number of positive: 127563, number of negative: 512437\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.207201 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4036\n",
      "[LightGBM] [Info] Number of data points in the train set: 640000, number of used features: 43\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.199317 -> initscore=-1.390567\n",
      "[LightGBM] [Info] Start training from score -1.390567\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[1000]\tvalid_0's auc: 0.721464\n",
      "[2000]\tvalid_0's auc: 0.723649\n",
      "Early stopping, best iteration is:\n",
      "[2714]\tvalid_0's auc: 0.724144\n",
      "[0.7243989768290735, 0.720405047892458, 0.724143719768038]\n",
      "************************************ 4 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] Number of positive: 127635, number of negative: 512365\n",
      "[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.210171 seconds.\n",
      "You can set `force_col_wise=true` to remove the overhead.\n",
      "[LightGBM] [Info] Total Bins 4029\n",
      "[LightGBM] [Info] Number of data points in the train set: 640000, number of used features: 43\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.199430 -> initscore=-1.389863\n",
      "[LightGBM] [Info] Start training from score -1.389863\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[1000]\tvalid_0's auc: 0.720496\n",
      "[2000]\tvalid_0's auc: 0.722882\n",
      "[3000]\tvalid_0's auc: 0.723572\n",
      "Early stopping, best iteration is:\n",
      "[3489]\tvalid_0's auc: 0.723692\n",
      "[0.7243989768290735, 0.720405047892458, 0.724143719768038, 0.7236917059085686]\n",
      "************************************ 5 ************************************\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] Number of positive: 127698, number of negative: 512302\n",
      "[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.068304 seconds.\n",
      "You can set `force_row_wise=true` to remove the overhead.\n",
      "And if memory is not enough, you can set `force_col_wise=true`.\n",
      "[LightGBM] [Info] Total Bins 4021\n",
      "[LightGBM] [Info] Number of data points in the train set: 640000, number of used features: 43\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.199528 -> initscore=-1.389246\n",
      "[LightGBM] [Info] Start training from score -1.389246\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[1000]\tvalid_0's auc: 0.720249\n",
      "[2000]\tvalid_0's auc: 0.722772\n",
      "[3000]\tvalid_0's auc: 0.723502\n",
      "[4000]\tvalid_0's auc: 0.723767\n",
      "Early stopping, best iteration is:\n",
      "[4474]\tvalid_0's auc: 0.723835\n",
      "[0.7243989768290735, 0.720405047892458, 0.724143719768038, 0.7236917059085686, 0.7238346328989569]\n",
      "lgb_scotrainre_list:[0.7243989768290735, 0.720405047892458, 0.724143719768038, 0.7236917059085686, 0.7238346328989569]\n",
      "lgb_score_mean:0.7232948166594191\n",
      "lgb_score_std:0.001465571038410448\n"
     ]
    }
   ],
   "source": [
    "import lightgbm as lgb\r\n",
    "\"\"\"使用lightgbm 5折交叉验证进行建模预测\"\"\"\r\n",
    "cv_scores = []\r\n",
    "for i, (train_index, valid_index) in enumerate(kf.split(X_train, y_train)):\r\n",
    "    print('************************************ {} ************************************'.format(str(i+1)))\r\n",
    "    X_train_split, y_train_split, X_val, y_val = X_train.iloc[train_index], y_train[train_index], X_train.iloc[valid_index], y_train[valid_index]\r\n",
    "    \r\n",
    "    train_matrix = lgb.Dataset(X_train_split, label=y_train_split)\r\n",
    "    valid_matrix = lgb.Dataset(X_val, label=y_val)\r\n",
    "\r\n",
    "    params = {\r\n",
    "                'boosting_type': 'gbdt',\r\n",
    "                'objective': 'binary',\r\n",
    "                'metric': 'auc',\r\n",
    "                'learning_rate': 0.01,\r\n",
    "                'num_leaves': 14,\r\n",
    "                'max_depth': 19,\r\n",
    "                'min_data_in_leaf': 37,\r\n",
    "                'min_child_weight':1.6,\r\n",
    "                'bagging_fraction': 0.98,\r\n",
    "                'feature_fraction': 0.69,\r\n",
    "                'bagging_freq': 96,\r\n",
    "                'reg_lambda': 9,\r\n",
    "                'reg_alpha': 7,\r\n",
    "                'min_split_gain': 0.4,\r\n",
    "                'nthread': 8,\r\n",
    "                'seed': 2020,\r\n",
    "                'silent': True,\r\n",
    "    }\r\n",
    "    \r\n",
    "    model = lgb.train(params, train_set=train_matrix, num_boost_round=14269, valid_sets=valid_matrix, verbose_eval=1000, early_stopping_rounds=200)\r\n",
    "    val_pred = model.predict(X_val, num_iteration=model.best_iteration)\r\n",
    "    \r\n",
    "    cv_scores.append(roc_auc_score(y_val, val_pred))\r\n",
    "    print(cv_scores)\r\n",
    "\r\n",
    "print(\"lgb_scotrainre_list:{}\".format(cv_scores))\r\n",
    "print(\"lgb_score_mean:{}\".format(np.mean(cv_scores)))\r\n",
    "print(\"lgb_score_std:{}\".format(np.std(cv_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "[LightGBM] [Info] Number of positive: 127698, number of negative: 512302\n",
      "[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.110340 seconds.\n",
      "You can set `force_row_wise=true` to remove the overhead.\n",
      "And if memory is not enough, you can set `force_col_wise=true`.\n",
      "[LightGBM] [Info] Total Bins 4021\n",
      "[LightGBM] [Info] Number of data points in the train set: 640000, number of used features: 43\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.199528 -> initscore=-1.389246\n",
      "[LightGBM] [Info] Start training from score -1.389246\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[1000]\tvalid_0's auc: 0.720249\n",
      "[2000]\tvalid_0's auc: 0.722772\n",
      "[3000]\tvalid_0's auc: 0.723502\n",
      "[4000]\tvalid_0's auc: 0.723767\n",
      "Early stopping, best iteration is:\n",
      "[4474]\tvalid_0's auc: 0.723835\n",
      "调参后lightgbm单模型在验证集上的AUC：0.7238346328989569\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAH1CAYAAABvDS3zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XdgU+X+x/F3ku69d6FQyh4tewplLwFFRMSNqCigKCLqvepdKkMUFbdXRMWBiiAbAVEEZJXdMkqB7pnukSY5vz/4yb1cGSk0PUnzff2l6WnyySFNPjnnOc+jURRFQQghhBAOR6t2ACGEEEKoQ0qAEEII4aCkBAghhBAOSkqAEEII4aCkBAghhBAOSkqAEEII4aCkBAghhBAOykntAEII2zNw4EAKCgrQ6XR4eHjQr18//vrXv+Lp6QnAgQMHeOONNzhy5AharZZu3boxe/ZsWrRocfE+ysvLWbx4MZs3b6akpITAwEASExOZNm0aAQEBaj01IcR/kSMBQojLeu+990hKSuKHH37g+PHjfPDBBwAkJSUxZcoUBg0axK+//sqWLVto1aoVkyZNIj09HQCDwcC9997L6dOn+eijj9i/fz9ff/01fn5+HDlyRM2nJYT4L3IkQAhxVcHBwfTt25fk5GQAFixYwNixY7n33nsvbjNr1iyOHTvGW2+9xfz581m1ahXZ2dksW7bs4tGDwMBAHnvsMVWegxDi8uRIgBDiqnJycvj1119p0qQJVVVVJCUlMXz48D9tN2LECHbu3AnAzp076dev38UCIISwTXIkQAhxWX98a6+srKRnz57MnDmTkpISzGYzwcHBf9o+ODgYvV4PQHFxMe3atWvQvEKIupMjAUKIy1qyZAlJSUl89tlnnDlzBr1ej4+PD1qtlvz8/D9tn5+fj7+/PwB+fn6X3UYIYVukBAghrqp79+7ceuutzJs3Dw8PD+Lj49mwYcOftlu/fj09e/YEoHfv3uzYsYPKysqGjiuEqAMpAUKIa7r33nvZuXMnKSkpPPXUU/zwww8sW7aM8vJySkpKeP311zl48CDTp08HYOzYsYSFhTFjxgxSU1Mxm83o9Xree+89tm/frvKzEUL8QUqAEOKaAgICGDt2LEuWLKFr16589NFHbN68mX79+pGYmEhycjLLly8nJiYGABcXF5YuXUrz5s154IEH6NKlCxMmTECv19OxY0d1n4wQ4iKNoiiK2iGEEEII0fDkSIAQQgjhoBqkBMybN4+BAwfSqlUrTp48edltTCYTf/vb3xg8eDBDhgxhxYoVDRFNCCGEcFgNUgIGDRrEF198QWRk5BW3+fHHHzl//jybNm3i66+/5q233iIjI6Mh4gkhhBAOqUFKQNeuXQkPD7/qNuvWrWPChAlotVoCAgIYPHjwZS9DEkIIIUT9sJkxAdnZ2URERFz8//DwcHJyclRMJIQQQjRujW7aYL2+ArNZLniwlsBALwoLy9WO0ejJfrY+2cfWJ/sYqmqM5BdXkV9SRUmZAYPRTGFpNcVlNWQVVVBba77s7+m0Gvx93HB10hEW6I67qzMmk5nQAA98XLV4bFuFa0kBHV/91w3ls5kSEB4eTlZW1sVriP/3yIClzGZFSoCVyf5tGLKfrU/2sfU5wj6urK4lu7CS1MwScvVVmMwKZ7NLKauqRV9W86ftg/3cCPB2I9zfAzQagnzciAn3JtTfAy93Z3y9XHDSXflAvamqiszcLLwHDb7h7DZTAoYPH86KFSsYOnQoxcXF/PTTT3zxxRdqxxJCCCEAKK00kJFXzonzxSSf11NVY6S4rIaKauOftm0S6kWovzsDEiIJ8XPHz8uFQF83/Lxcr/oBfzXmmguFQufuTvQzz6Fzcb6h5wMNVAL++c9/smnTJgoKCrj//vvx8/Nj7dq1TJ06lZkzZ9KhQwfGjh3LoUOHGDp0KHBhBbPo6OiGiCeEEEJcVFxew9EzRejLaygsqeZURjGVNUZKyg2XbOfr5UKH2ECahHjj7eFMTJg3If7uODvp6j2TubqazLfeQOPsTOTjT6Jxqp+P70Y3Y2BhYblDHH5SS3CwN/n5ZWrHaPRkP1uf7GPrs4d9XFZpIC27jKyCCpLP6cktqiSvuOriz73cnQn2c8PVWUf75oGE+rvTNNSbQF83NBpNg2Q0VVWRuXgR1WdSCZsyFZ8evQDQajUEBnrd0H3bzOkAazKZjOj1+RiNhmtvLK4qL0+L2Xz5gSz2zMnJBX//YHQ6h/iTEMIhKYpCel45B07mczanjOzCCvKLqy/+3NPNiRB/D25PiKRFlC9RwZ64uaj7nmCqrCDzjdeoPneO8Iem4d21W73ev0O84+n1+bi5eeDpGdZgza2xcnLSYjQ2rhKgKAoVFaXo9fkEBV19PgshhH0pqzSQdKqAtOxSjqUVUVDynw/9IF83+sdHEN8iiGYRPni7O9vcZ0T2B+9Tfe4cEY88hldC53q/f4coAUajQQqAuCKNRoOnpw/l5cVqRxFC3ACzopCRV05WQQUZ+RWczijmZEbJxZ/HtwhiRM+mdGgeQKBPwx3OvxHBt03AWDwYz/bWWX3TIUoAYBf/2EI98voQwv4oisK53DKOpRVx/Kye5HP6iz/TajSEBXowsmdTWkb70a6ZPzqtzcyPd1XGkmLK9u7Bb9AQXKOicY2y3iB5hykBQggh7F9phYGU83oOnS4k6VQ+1QYTwIXL8eIjCPZ3p2W0H01CvHF2so8P/f9Wq9eTsXAexmI9Xp0ScA4OturjSQloYE89NZN+/W5i3LjbLt6mKAq33z6O5557gYSELlf83enTH2LSpLvp06ffZX9+9mwad901gRkzZjFx4uSLt//rXy/RunUbxo+fePG2t99+A3d3d6ZMeRiA5ORjvP/+ErKyMvHx8cXDw4MHHniI+PgbPwdlMpl4442F/P77TjQaDXfddR833zzuT9tt2/YTn37674v/n5+fS6dOnXn55QUcOXKIJUsWU15+YaRxr159efTRmWg0Gk6dOsErr/wds1nBaDTSoUMnZs16GhcXF8xmM2++uYh9+35Hq9USFBTCc8+9QFCQdf+whBD1w1Br4nxeOUmn8tl1NIfi/79Mz8vdmVbRfrRs4kevdmH4ebmqnPTG1RYWkLFwHqayMqKemG31AgBSAhrcqFFj+Oqrzy8pAUlJ+9FqNTf8gbt27Wq6dOnGunU/XlICriU19TRPP/0Ef/3r3+nx/5eeZGZmcOrUiRvK84dNm9aTmZnOV1+tpKSkhAcemEzXrt0JD790RsjExMEkJv5nBqz777+TIUOGAeDp6cnzz79EdHQTDAYDjz8+jY0b1zF8+CiaNGnK++8vxdnZGbPZzF//+gyrVn3PhAl3sGPHLxw/fpSlS7/EycmJt95axNKlHzN79tx6eW5CiPqXU1TJ/hN57DiSQ56+kj8uZG8W7kN8XDBdWwXTMtrvuifdsUWG/DwyFs7DXFlJ5JNP4948tkEe1+FKwG9HstlxONsq9923Yzh9Olx9dHm/fv157bVXOHs2jZiYZsCFD++RI29Go9Gwb98ePvzwXQyGGkwmE/fc8wCDBw+75mMbjUY2bVrHkiUfMXv2TJKTj9GmTTuLcn/xxaeMHj32YgEAiIyMIjIyyqLfv5atWzdz883j0Gq1+Pv7069ff7Zt+4k777znir9z4kQK+fl59O3bH4DmzVtc/JmLiwstW7YiJ+fCv6Orq9vFnxmNRmpqatBqL5zj12igttaAwWBAq9VSWVn5p/IhhFCXWVE4l1PGgZP5HDlTyPncC+sNRId40a9jBO2aBRAb4UOAj9s17sl+1aSnoxgMRM1+BremMQ32uA5XAtTm7OzMkCEjWLduNY8++jiVlRX8+ut2Pv/8GwBatmzNO+98hE6no6iokClT7qZ79174+Phc9X537dpBVFQToqKiGTFiNGvXrra4BJw8mcKAAYMs2jYt7Qx//etzl/1Zt249eOyxx/90e25uDmFh/ylHoaFh5OXlXvVx1q5dxZAhI3B2/vO0mHp9ET//vJUFC964eFtBQT6zZz9OZmYGvXr1YcyYWwHo0+cmkpL2M3bsMFxd3WjaNIYnn3zGoucqhLCeiupaTp4v5lRmCXuT8ygsrUYDNI/wYXz/5vRoE0qQn7vaMa3OXGtA6+yCd+cueLZti9atYZ+zw5WAPh2u/W3d2kaNGsPs2TN4+OHpbNmymQ4dOhESEgpAcbGeV175OxkZ59HpnCgtLeH8+XO0b9/hqve5du1qRowYDcDw4aO4//7JzJjxJK6urlcc+X49I+KbNWvO0qXL6/x7dWEwGNi8eSNvvfX+n35WWVnBM888yR133EXLlq0v3h4UFMzSpcupqqri73//K9u3b2Xw4GGcPJnC2bNnWblyPR4eHixevJC33lokRUCIBlZTa+JMZgnHz+n5/Xjuxev1tRoNLSJ9GNW7KZ1ig/D3tv9z+5aqyUgnc/HrhN5zH54dOjZ4AQAHLAG2IC6uJYGBwezevZN161YzYcKdF3/22muv0qfPTbz88gI0Gg133HErBsOfV6H6b0VFhezZs5uTJ0+wdOlHAFRXV7N9+1aGDh2Bn58fJSUll/xOSUnxxcP9LVu2Jjn5GDfdNOCa2a/nSEBoaBg5OdkXj0z875GB//XLL9uIiIikRYu4S26vrq5mzpxZdO/ek0mT7rrs77q7uzNo0BA2bdrA4MHDWLfuR7p06YqX14WpNYcOHckrr/z9ms9TCHHjKqtr2X0sh4OnCzicWki1wYRWoyEm3Jv4uCASWgQRG+mLi3P9z7Vv66rPnyNj0QI0Tk4NMgDwSqQEqGTUqDH8+98fkJubTb9+/S/eXlZWRnh4OBqNhr17d5OZmX7N+9qwYS0DBgzihRf+cfG2zZs3sGbNKoYOHUG3bj1YvHgRt912Bz4+PuTm5rBnzy7uvvs+ACZNuptZsx6lc+eudOvWA4CsrMzLnia4niMBiYmD+fHHH+jffyAlJSX8+ut2liz58Irbr127mlGjxlxyW01NDc88M4u2bdvz4IOPXPKzzMwMgoNDcHFxoba2ll9/3U5s7IUxBOHhkezdu5uJEyfj5OTE7t2/0byBBtwI4Yjyiqs4eqaQ5HN69p/IB8DHw5nOLYPp3iaE5hG+eLnf+Op39qw67QwZry9E6+ZO1OxncAkJUS2LlACVDBkynCVLFjNmzC2XnPeeNm06r702j48//oA2bdoSGxt3lXu5YN26H3nssScuua1fvwEsXPgK2dlZdOvWk3HjbmXGjIfRaDRotVqeeOJpmjSJAS4cmXj11df58MN3WLDgZdzc3PDz82fKlEcu82h1N2zYSI4fP8odd9wCwH33PUhERCQAP/zwLQUFBRc/2HNzczhy5BB///url9zHmjWrSEraT0lJCXv27AYgMXEQ9947haNHD/PFF5+i0Wgxm03Ex3fhvvumAHDrrRNIS0vl3nvvQKfTERoaxpw5z9fL8xJCXLjEOSO/gt+OZJOaWUJqVikAfl4uDOvZlFZRvnSMDUQrE3IBUJufT8aiBeg8vYiaPQdnlS9XdohVBHNyzhEW1lSlRI1LY1w74A+29Dqxh9XX7J3s4xuTVVDB3pQ8dh/PJbeoEgB/b1cGJETStVUw4YGeso8vQ1EUitb+iE/vPjgHBN7QfckqgkIIIRpEtcFI8lk9R9KKOJ1RTEZ+BXDh2v27h7UivoVjDeqrq8qUZJz8/XEJDSNw9Jhr/0IDkRIghBDiilKzSti6P4O9KXkYTQquLjqahnhxx8AWdG0d0qiv3a8vFUcPk7XkLdxbtSHqiSfVjnMJKQFCCCEukaev5MDJAvafyCM1qxSdVkOPtqF0bR1C+2YBjWqmPmsrP3SQ7HffxiU8gvApU9WO8ycOUwIURZGV4sQVNbKhMULUWV5xFbuP5bDr2H/O8UcFezJhQCwDEiJxd3WYj4t6U3ZgP9nvv4NrdBOiZs1G5+mpdqQ/cYh/VScnFyoqSvH09JEiIP5EURQqKkpxcnJRO4oQDcpoMnPwVAE7j+Zw6HQBChAb4UPioDg6tQgk1N9D7Yh2S1EUirdsxi2mGZGPP4nOwzb3pUOUAH//YPT6fMrLi9WOYve0Wi1mc+O7OsDJyQV/f1lZUDR+ZkUhNbOETXvSOZlRTFllLV7uzozs1ZS+HcPlg78eKGYzGq2WiOmPo9GgykyAlnKIEqDTOREUpO5UwY2FXPIjhH2qqjHy88FMtuzPoKi0Biedlubh3kwZ1ZS2MXKev76U7PiF0l07iZw5C5277X74/8EhSoAQQjiqPH0lK35O5ciZQgy1ZmIjfRh/UyydWgTh4SYfAfWp+Oet5H2+DI927S8sYWoH5BUghBCN0LmcMjbtTWf38RxQoG2zAEb3akqrJv5qR2uU9D9tJv+rL/Ds2InwaY+hdbaPMUZSAoQQopGoNZpJOpXPL4eyOH5Wj4uzlkGdoxjaLdohluVVS/G2reR/9QVenbsQ/tA0NE7289FqP0mFEEJc1vncMn49lM3elFxKK2vx8XRhXL9mDOwc5fCL9TQEj9at8U0cSMjEO+2qAICUACGEsEuKopCeV873v5zhcGohOq2Gds0C6B8fQcfYQHRaGehnTYqiUHn0CB7tO+ASHkHo5HvUjnRdpAQIIYQdKa0wsOtYDtsOZJJXXAXA2L7NSOwciY+HfZyHtneKolDw3Qr0G9YR/shjeHftpnak6yYlQAgh7EBJeQ0b96Sz9UAGBqOZpmHe3DW0Je2bBxIi5/sbjKIo5H/9JcU/bcJ3wEC8OndRO9INkRIghBA2ymQ2c/h0IXtP5LEvJR+jyUy31iGM6tWUJqHeasdzOIrZTN6Xn1OybSt+g4cQPPFOu5+FVkqAEELYGEVROJxayPKfTpJfXA1An/ZhjOjZlIgg25t/3lHUpJ+n5Jft+A8fSdD4CXZfAEBKgBBC2Iyi0mp2HMlm55Ec8oqrCPFz5+Ex7ejUIhA3F3m7Vptb0xiavvA3XCIiG0UBACkBQgihKrNZ4WhaETsOZ7HvRD4AcVG+jOzVlF7tQnF20qmc0LEpJhM5Sz/GKz4B7y7dcI2MUjtSvZISIIQQKiivqmXL/gw27T1PVY0JNxcdg7pE0b9TBFEhXmrHE4BiNJL94XuU79+Ha1S02nGsQkqAEEI0oJIKA+t3n2NbUia1RjORwZ4M6hJFn/Zh8q3fhphra8l+bwkVhw4SPHES/kOGqR3JKqQECCFEAyirNLBxTzpbDmRgqDXRvU0oI3o0kVH+NkgxGsla8iaVR48QMvke/BIHqh3JaqQECCGEFRlNZrbuz2DVb2lU15joGBvIbYktiJRR/rZLp8M1MgrvLl3x7ddf7TRWJSVACCGs5GR6MZ9tPEFmQQXNI3yYPKQlzcJ91I4lrsBcXY2xuBiXsDCCJ0xUO06DkBIghBD17MR5PWt2nuXYWT2+Xi48Oq49XVoFN5rLyhojU2UlmYsXYSwqJOZf89C6OMYUzFIChBCinuQXV7F880kOpRYCcOtNzRncNUqu8bdxpooKMl5fSE36ecIfesRhCgBICRBCiBuWWVDBul3n2H08B51Wy7h+zRjcJRoPN3mLtXWmsjIyXl+IISuTiGnT8YpPUDtSg5JXqBBCXKfzuWVs2pvO7mO5aLUwMCGKET2bEODjpnY0YaGC1SsvFIDpM/Fs31HtOA1OSoAQQtSRoiis2pHG6t/O4uqso3f7MMb1ayYf/nYo+LaJ+PTsjXtsC7WjqEJKgBBC1EFadikrtp0m5XwxCXFB3D+yDV7uzmrHEnVQq9dT8N03hN51D1o3d4ctACAlQAghLFJTa2LT3nRW/ZqGq4uWSYPiGNw1Skb825nawgIyFs7DVF6OYehw3Jo0VTuSqqQECCHENSSfLeKjtcnoy2roFBvIgze3xdNNvv3bG0N+HhkL5mGuriJy1tMOXwBASoAQQlyWoiicTC9m1Y40Us4X4+vlwszxHenUIlC+/dshQ24OGQvnYTYYiHpqDm5NY9SOZBOkBAghxH8xmxUOnS5gw57znMoowdfThfH9m9M/PlLO/dszjRadtw+R9z+Ia3TjXBHwekgJEEIILszxf+BkPqt/O0tWQQWBPm5MSIwlMSFSJvuxY7WFhTgFBOASEkKTv74kR3H+h7yyhRAOrbK6li0HMtl+MJOi0hqCfN2YMqoNPduFotNq1Y4nbkD1ubNkLFqA38DBBI29RQrAZUgJEEI4JEVR2HYgg9W/naWkwkCbpv5MGhRHQlwwWq18WNi7qjNnyHxjIVo3d3x69VE7js2SEiCEcDhZBRXMW57EifN6ooI9eezWDrSI9FU7lqgnVadPkfnGa+i8vYma/QzOgUFqR7JZUgKEEA7DaDKz9UAmK385g5OTlnuGtaJ/fIQcJm5EzNVVZL71BjpfX6KeegbngAC1I9k0KQFCiEZPURSOpRXx5ZZTZBdWEhfly3P390CpNaodTdQzrZs74Q9NwzUyCic/P7Xj2DwpAUKIRu1cThnLNp4gLbsUPy8XHh3Xns6tggnycyc/v0zteKKeVBw5jKmiHJ+evfFs117tOHZDSoAQolGqqjGy4ffzrNt9Dk93ZyYPaUm/juG4OOvUjibqWfnBJLLfW4JLVDTe3Xuikas6LCYlQAjRqCiKwtYDmXz/yxmqaox0bhnMvcNb4e3honY0YQVl+/eS/cF7uDVpSuQTT0kBqCMpAUKIRiO/uIplG09wLK2IuChfxvVrTusmfjLwr5Eq/X03OR9/gFuz5kQ+8RQ6d3e1I9kdKQFCCLtXUmHgy59Osjc5DycnLRMHtmBIt2i08uHfqBlysnFvEUfkzFlo3dzUjmOXpAQIIexWTa2JLfszWLPzLLVGM0O7RzOkazQBPvKB0JiZKivQeXgSOGYcmExonOSj7HrJnhNC2J1ao5l9J/JY+csZCkqqad8sgAmJLYgO8VI7mrCy4m1bKFz9A9Fzn8clNAykANwQ2XtCCLuhKAp7U/L4/pcz5OmrCPJ1Y/Yd8bRp6i/n/R2AfvNG8r/+Es/4BJwCAtWO0yhICRBC2AWjycyyDSfYcSSbIF+3C9f7t5R5/h1F0fq1FHy3Aq8uXQmf+oicAqgnsheFEDbNbFY4fKaQ735OJbOgghE9mnBr/+aywp8DKf19NwXfrcC7e0/CpkxFo5O5HuqLlAAhhM06m1PKx2uSySyoIMDHlcdu6UCXVsFqxxINzCuhM0ETJuI/ZJjMA1DPpAQIIWyOvqyGVTvOsONwDt4ezjw4ug3dWofi7CQfAI5CURSKt/yET+/e6Dw8CRg2Qu1IjZKUACGETTmdUcLbK49QVmGgb8dwJiS2wMvdWe1YogEpikL+18sp/mkzmE34Dx2udqRGS0qAEMIm1BrNrP4tjY17zuPv7crsKd2JCpZL/hyNYjaT98VnlGzfht/gofgNGaZ2pEZNSoAQQnXnc8t4f/UxsgsrSYgLYvKQljLhjwNSzGZyl31C6Y5f8R8+kqDxE+TSTyuTEiCEUE21wcjq386yaU86nu5OPDquPV1aBcsbv4MylZdTmZJMwM1jCRwzTl4HDUBKgBCiwSmKws6jOXz/yxn0ZTX0aBvKpMFx+MhKfw5JMZlAo8HJx4emL/wNnYen2pEchpQAIUSDKq+qZdmGFPadyCcyyJMpo9rQNiZA7VhCJYrRSPb776L19CT03vulADQwKQFCiAZz8HQBn65PobTSwOjeMYzr10xW+nNg5loD2e8uoeLwIYLvuFMO/6tASoAQwupqDCa+/TmVLQcyCAvw4LFbO9Ai0lftWEJFZoOBrCVvUnnsKCF33YPfgIFqR3JIUgKEEFZTazTz0750Vu88S43BxKAuUdye2EIm/RFkv7eEyuPHCL3vAXz73qR2HIclJUAIYRWZ+eW888NRsgsrad8sgDF9mtEiSr79iwv8Bg3Bu1sPfHr1VjuKQ5MSIISoV2ZFYcu+DL7bnoqLs47pt3YgIS5IzvcKTJWVVJ1IxiuhC57t2qsdRyAlQAhRj06c17Ns4wmyCytp3cSPqTe3w9/bVe1YwgaYKirIeH0hhox0Yl6ej3OAXBFiC6QECCFu2B9T/q7ddQ5/b1ceGduObq1D5Nu/AMBUVkbGogUYsrMInzZdCoANabASkJaWxty5cykuLsbPz4958+YRExNzyTaFhYU8++yzZGdnYzQa6dGjB3/5y19wcpKuIoStOpVRzFdbTpGWXUbfDuFMHtISVxdZ711cYCwpIWPRAmrzcomY/jie7TuoHUn8lwYbovviiy9y5513snHjRu68805eeOGFP23z3nvvERsby48//sjq1as5duwYmzZtaqiIQog6MCsKK7ad5pXPD5CWXcZDY9rywKg2UgDEJcoPJlGbn0fkzFlSAGxQg3zFLiws5Pjx43zyyScAjB49mn/84x8UFRUR8F+HhTQaDRUVFZjNZgwGA7W1tYSGhjZERCFEHejLavjwx2OknC+mS8tg7h3RWpb7FZdQFAUAv/4D8GzfHufAIJUTictpkBKQnZ1NaGgoOt2Fbwg6nY6QkBCys7MvKQGPPvooM2bMoG/fvlRVVTF58mS6dOlSp8cKDJSlR60tONhb7QgOwVb384ZdZ/lo9VFMJoVHx3dkeK8Yuz33b6v72N5V5+aR8up8PKY/SnBsc5D9bLNs6mT7hg0baNWqFZ9++ikVFRVMnTqVDRs2MHz4cIvvo7CwHLNZsWJKxxYc7E1+fpnaMRo9W9zPZkVh8950vt56mrgoX+4b0ZrwQE8KCsrVjnZdbHEfNwaGvDwyFr6KuboaxWyWfWxFWq3mhr/4NsiYgPDwcHJzczGZTACYTCby8vIIDw+/ZLvPP/+cMWPGoNVq8fb2ZuDAgfz+++8NEVEIcRX6shrmL0/i662naRfjz6zbOxEeKAu9iEsZcrJJn/8yZoOBqNnP4B3XQu1I4hoapAQEBgbSpk0b1qxZA8CaNWto06bNJacCAKKiovjll18AMBgM7Nq1i7i4uIaIKIS4ghPn9bzw8e+kZpZw19CWzJoYj5uLTR1EFDbAkJ9H+vxXwGQm+um5uDVpqnYkYQGN8sfoDStLTU1l7ty5lJaW4uPjw7x582jevDlTp05l5syZdOjQgfPnz/NAx8c7AAAgAElEQVTiiy9SUFCAyWSiR48ePP/883W6RFBOB1iXHEJtGLawn2uNZr7bnsqmvekE+rgy/daONA1rPOd2bWEfNybm2lryvlhGwLARuIRHALKPra0+Tgc0WAloKFICrEv+qBuG2vu5oLiKd1cdJS27jF7tQrlzSEs83RrX6H+193FjUX3+HM7+Aei8/1wQZR9bV32UADmmJ4S4RFp2KYtXHKKm1syj49rTtXWI2pGEjapKPU3mG6/h3qo1kdMfVzuOuA5SAoQQF+1LyeOT9cl4uDrx9J2diQySwX/i8qpOnSRz8SJ03j6ETLpL7TjiOkkJEEJgNJlZtvEEOw5n0zTUm0dvaU+wn7vasYSNqkxJJvOtN3Dy9yfqqWdw9vdXO5K4TlIChHBwpZUGPl6TzJEzhQzv0YRb+jXH2anBZhQXdkYxm8n7ajnOgYFEPTUHJ18/tSOJGyAlQAgHlllQwRvfHKKkoobJQ1oyqEuU2pGEjdNotUTOfAKNszNO3j5qxxE3SEqAEA5qX0oeH69Nxkmn4amJ8bRqIod0xZWVJx2g/FASoffcj3NAoNpxRD2REiCEg6mqMfLtz6lsS8qkSagX02/tQJCvnP8XV1a2bw/ZH76PW5OmKIYaNG7yemkspAQI4UDO5ZSxZOURCkuqGRAfwcSBcbL0r7iq0t93kfPRB7jFtiDy8SfRSgFoVKQECOEgdh/L4d/rUvBwc2LOnQly+F9cU+mu38j590e4t2xF5Iwn0Lq5qR1J1DMpAUI0clU1RpauT2FvSh6xkT5Mv6UDvl6uascSdsApIBDPTvGET30Erau8ZhojKQFCNGLncsp4b/Ux8vSVjO7dlJt7N5PL/8Q11WRm4BoZhUer1ni0aq12HGFFUgKEaITMisJPe9P5dvsZPN2cmDWhE+2by4hucW36TRvJ/+ZLIh9/Es8OHdWOI6xMSoAQjYzJbObTDRdm/+sYG8j9I9vg6+midixhB4rWraHg+2/x6tIVjzZt1Y4jGoCUACEakaLSapZuSOHomSJG9GjCbQNi0Wg0ascSNk5RFIrWrKZw1Uq8e/Qk7IGpaHRy1YgjkBIgRCNRUFzFK18coKK6lokDWzCsexO1Iwk7UZ16msJVK/Hp3ZfQ+x5Ao5VxI45CSoAQjcCuYzks23gCxazw9KQEYiN91Y4k7Ih7izgin3gKj7btpAA4GPnXFsKOKYrCht/P8+GPxwkP8OClB7pLARAWURSF/G+/oSr1NACe7TtIAXBAciRACDtVbTDywerjHDxdQHyLIB4e005m/xMWUcxm8j5fRskvP6Nx0uEe20LtSEIlUgKEsEOlFQbe+v4wZ7JKmZAYy7DuTdDKAEBhAcVsJnfpvynduYOAkaMJHHur2pGEiqQECGFnistrWLziMJkF5Twwsg19OoSrHUnYCcVkIuffH1H2+y4Cx4wj4OaxcvWIg5MSIIQdKakw8Mrn+ykqreHRW9qTEBesdiRhZxRjLUG33kbAyNFqRxE2QEqAEHaiptbE298dpqi0htl3xMsCQMJi5tpazNVVOHn7EP7wozIAUFwkrwQh7EBldS2LVxziTFYpU29uKwVAWMxcayD73bfJWDAPc22tFABxCTkSIISNy9NXsujrQ+SXVDFldBu6twlVO5KwE+aaGrLeeYvKY0cJufs+tM7OakcSNkZKgBA27FxOGa99fRCTWeHpOxJo3VSOAAjLmGtqyHzrDapOpBB63xR8+/ZTO5KwQVIChLBR53MvFABXZy1PTOhEZLCX2pGEHcn78nOqTqQQNmUqPj17qx1H2CgpAULYoG3703n3u0O4OOt4cmI84YGeakcSdiZo3Hi84jvjFZ+gdhRhw2SEiBA25pdDWSxafoBgP3eev7uLFABhMVN5OQUrv0MxmXDy85MCIK5JjgQIYUN2H8/h0/UpxLcMZtqYtjg7yTTAwjLGslIyFy3AkJ2NV0Jn3GKaqR1J2AEpAULYAJPZzJc/nWLrgUxiI32Ye083Ksur1Y4l7ISxpISM1+ZTm59HxIwnpAAIi0kJEEJlRpOZd384StKpAoZ0jWZCYiye7s5SAoRFjMV6MhbOp7aokMjHn8SjdRu1Iwk7IiVACBWlZZfy2cYTnM0pY3z/5ozqFaN2JGFnaov0mKqriJo1G/e4lmrHEXZGSoAQKjl4qoD3Vh3FxVnHlFGyEJCoG1NVFTp3d9ybN6fZK/PROruoHUnYIbk6QAgVHE4tZMnKI4QFePC3B7pLARB1YsjN5dyLz1O8bSuAFABx3eRIgBANbPfxHD5ek0xYgAezJyXg5S5TuQrLGbKzSF84H0wm3GJj1Y4j7JyUACEa0K+Hs/hkXQrNI3yYdXsnPN2kAAjL1WRmkLFwPmgg6ulncI2MUjuSsHNSAoRoIPtS8li6LoXWTfx4fEInXJ1lDgBhOVNFBRkL54FWR/TsObiER6gdSTQCUgKEsDKzorBu1zm+/+UM0SFePDGhEy5SAEQd6Tw9CRo/Afe4VriEykqSon5ICRDCihRF4cufTrFlfwYJcUFMvbmtFABRJ1Wpp1GMRjxatca3701qxxGNjFwdIISVGE1mPl6bzJb9GQxIiGT6rR1wc5HeLSxXdeokGYsWkv/VchSzWe04ohGSdyQhrKDGYOLtlUc4llbE6N4x3NKvGRqNRu1Ywo5UpiST+ebrOAUEEPn4LDRa+c4m6p+UACHqWbXByGtfHyQ1s5TJQ1oyqIuM4BZ1U3HsKFlvL8Y5OISop+bg5OurdiTRSEkJEKIe1RhMvP39Ec5klfLI2HZ0byMDuETdle39HZewMCKffBonbx+144hGTEqAEPWk1mjm/dXHOH5Wz91DW0oBEHWmGI1onJwIvfs+zDXV6Dw81Y4kGjk5ySREPaiqMbL420McPF3A5CEtSewspwBE3ZTt28O5v7+AsbgYjU4nBUA0CCkBQtygkgoDC75MIvmsnvtHtJYxAKLOSnfvJPv9d9F5eqF1c1U7jnAgcjpAiBtQUmHgn5/uo6SihodlDIC4DiU7fiX303/j3qo1kTOeQOsqJUA0HCkBQlynotJq3vzuMMXlNTw9KYGW0X5qRxJ2pmzvHnKXfoxHu/ZEPDpDCoBocFIChLgOGfnlzF+ehKHWxCNj20sBENfFo3Ub/AYPIWj8BFkOWKhCxgQIUUdp2aUs/OogGg385d6udGkVrHYkYWfK9u9DMRrReXsTcsdkKQBCNVIChKiDY2eLePmz/Wg08NTEeKKCvdSOJOxM4ZrVZL/7NsXbtqgdRQg5HSCEpfafyOfDNccI8Xdnzp2d8fWUb2/CcoqiULj6B4p+XIV3z174DRysdiQhpAQIYYltSZl8tvEETUK8mH5rBykAok4URaHg+2/Rr1+LT59+hN57v6wFIGyClAAhriH5nJ4vNp2kfbMAZozviLOTvHmLujHqiyj5eSu+/RMJmXy3FABhM6QECHEVOUWVvL/6wimAR8a2lwIg6kRRFDQaDc4BgTR94e84BQXJapLCpsg7mhBXkFlQwcuf7cdoNDNtXHs83KQzC8spZjN5ny2laN0aAJyDg6UACJsjJUCIy8jTV7Jg+QEA5k7uTHSIXAUgLKeYzeQu/ZiSX7Zjrq5WO44QVyRfbYT4H5n55cz/MgmzWWHOnZ2JkgIg6kAxmcj594eU/b6bwLG3EHjzWLUjCXFFUgKE+C+nM0tY8GUSTjqtHAEQdaYoCjkfvU/Z3j0E3XobASNHqx1JiKuSEiDE/8spquTt7w7j6ebEc3d3IcjXXe1Iws5oNBo82rTDrVlz/IcOVzuOENckJUAIoMZg4p2VRzCZFWbfkSAFQNSJudZATXoG7s2b43tTf7XjCGExGRgoHF6t0czrKw6RmV/B1JvbyRgAUSfmmhqy3lxMxmvzMJaUqB1HiDqREiAcmllRWLo+mZPpxUwaHEfH2EC1Iwk7Yq6uJvPN16lMOU7InXfj5OurdiQh6kROBwiHVWs0sXR9CruO5TKubzMGd41WO5KwI6aqKjIXL6L6TCphDz6ET49eakcSos6kBAiHVGMwsWTlEY6mFTGsezQ394lRO5KwMyU/b6U67QzhD03Du2s3teMIcV2kBAiHYzSZWfztIVLOF3Pn4Dg5AiCui/+wEXi0aYtbTDO1owhx3WRMgHAoRpOZxSsuFIDJQ1pKARB1YiwtJfPtxdQWFqLRaqUACLsnRwKEQ1mz8yzHzuq5e2hLEjtHqR1H2BFjcTEZi+ZTW1BAbX4ezoEyiFTYPykBwmF8/0sqa3aeo0urYCkAok5q9XoyFs7DWKwncuYsPFq3UTuSEPVCSoBwCLuP5bBm5zm6twnhwdFt1Y4j7EhtUSEZC+ZhKisl6omncI9rqXYkIeqNlADR6B09U8jHa5OJjfBhyqi2OOlkKIywnNbVDafAQMKmPox781i14whRr6QEiEYtV1/Ju6uOEujjxozbOuLsJAVAWKa2IB+dry86T0+inpqDRqNRO5IQ9a7O74iFhYXWyCFEvdOX1bDwy4No0DBrYid8PFzUjiTsRE1WFudf+Re5y5YCSAEQjZZFJaCsrIw5c+bQsWNHBg0aBMDWrVt58803rRpOiOt1NqeUlz7ZQ3l1LU/dEU+ov4fakYSdqMnMIGPBq6CYCRg+Su04QliVRSXgpZdewtXVlY0bN+Ls7AxAp06dWLt2rVXDCXE9yioNvPfDMbQaDc/f3YVm4T5qRxJ2ovr8OdIXvAo6LdFznsU1MlLtSEJYlUVjAnbu3Mn27dtxcXG5eFgsMDCQgoICq4YToq4URWHp+hQKS6t5elICUcGyIqCwjGIykf3+O2hdXIma/QwuISFqRxLC6iwqAV5eXpSUlBAcHHzxtuzsbIKCgqwWTIjrsXFPOkmnChjTJ4aW0X5qxxF2RKPTEf7wo+g8PHAOCr72LwjRCFh0OmD8+PE8/vjj7Nu3D7PZzOHDh3n22WeZOHGitfMJYbFfD2XxzbbTxLcIYkxfmc5VWKby5AkK1/4IgFuTplIAhEOx6EjAww8/jIuLC88//zw1NTXMnj2biRMnct9991n8QGlpacydO5fi4mL8/PyYN28eMTExf9pu3bp1vPvuuyiKgkaj4ZNPPpEjDuKa9qbksXR9Cq2i/Xh4bDu0MppbWKAy+TiZb72Bc0Ag/oOGoHVzUzuSEA1KoyiKcq2NioqKCAgIsPj2y7nnnnsYP348Y8eOZdWqVXz33XcsW7bskm2OHDnCM888w6effkpwcDBlZWW4uLjg6upq4dOBwsJyzOZrPiVxnYKDvcnPL1M7xiVSM0uYt/wAYQEePHd3F9xc7H/6C1vcz42NU/ppkl+eh3NIKFFPPo2Tr6/akRodeR1bl1arITDwxsY9WXQ6YPDgwZe9ffjw4RY9SGFhIcePH2f06NEAjB49muPHj1NUVHTJdkuXLuWBBx64OPbA29u7TgVAOJ7SCgPvrjqKn5crsyclNIoCIKyv/GASyf96FZewcKJnPyMFQDgsi94xL3ewoKKiwuIJNLKzswkNDUWn0wGg0+kICQkhOzv7kiMJqampREVFMXnyZCorKxkyZAjTpk2r00QdN9qKxLUFB3urHQEAk1nh9RU7Ka+s5ZXH+hLbxF/tSPXKVvZzY6Q4K3g2b0a7F/+Ck5e8Z1iTvI5t21VLwMCBA9FoNNTU1FycJOgPer2eYcOG1WsYk8nEiRMn+OSTTzAYDDz44INEREQwbtw4i+9DTgdYly0d3vtueypHUgu4e2hL/N2dbCZXfbCl/dyYGEtKcPL1RdOuMx379aWgqBKqZD9bi7yOras+TgdctQT861//QlEUpk2bxj//+c9LfhYUFERcXJxFDxIeHk5ubi4mkwmdTofJZCIvL4/w8PBLtouIiGD48OG4uLjg4uLCoEGDOHz4cJ1KgHAMvx7KYu2uc/RoG8qABJnQRVxb6a7fyP3sU6JmPY17XBya/z8yKYQju2oJ6NWrFwC//fYbXjdwyCwwMJA2bdqwZs0axo4dy5o1a2jTps2fBhWOHj2a7du3M3bsWIxGI7t37673ow3C/iWf0/PphhPERvjwwMjWMq+7uKaSHb+Q++knuLdqjWuTJmrHEcJmWDxZ0MmTJ9m3bx96vf6SMQLTp0+36IFeeukl5s6dyzvvvIOPjw/z5s0DYOrUqcycOZMOHTowatQojh49ysiRI9FqtfTt25fbbrvtOp6WaKxy9ZW8s/IIQb5uzLo9Hmcn+TYnrq5421byvliGR7v2RDw2E62LLCQlxB8sukRwxYoV/POf/6RXr1789ttv9OnTh127dpGYmMgbb7zREDktJmMCrEvNc3xVNUb+tnQvpRUG/nJPVyKCPFXJ0RDkXGr9qExJJmPhPDw7diJ82mNonf9TAGQfW5/sY+uy+piAP3z44Yd88MEH9OjRg27duvHee++xbds2Nm3adEMPLoSlao1mPvzxOHn6Kp6aGN+oC4CoP+6tWhNyz3349u6LxkkuHxXif1k0T0BBQQE9evS48AtaLWazmQEDBrBlyxarhhMCwGQ28+4PRzl4uoCJA1vQrpllE1QJx6X/aROGvDw0Gg1+Nw2QAiDEFVhUAsLCwsjMzASgadOm/Pzzzxw8eBAn+cMSDeCHX9M4eLqACQNiGdZdBnWJK1MUhYIfvif/q+WUbN+mdhwhbJ5Fn+L3338/p06dIjIykmnTpvH4449jNBqZO3eutfMJB3cyvZh1u87Rq10YI3o2VTuOsGGKolDw3Qr0G9bh0/cmgsZPUDuSEDbPohIwYcJ//pgSExPZu3cvBoMBb2+ZCUpYz9mcUhZ/e5gAH1cmD2mpdhxhwxRFIf/rLyn+aRO+AwYScuddaLQWHegUwqFd11+Jq6srRqOR1157rb7zCAFAVkEFr311EGedhicnxuPhJqeexJUpBgPVZ07jN3gIIZPvlgIghIWu+c66cuVKkpOTadq0KRMnTqSqqop33nmHr776is6dOzdERuFgSisNvPndYQBmT0ogPFCuBBCXp5jNKEYjWldXop56Bo2Li0weJUQdXLUEzJ8/n9WrV5OQkMDatWs5dOgQBw8epF27dixfvpw2bdo0VE7hIIwmM4u+OkhRaTWzJnQiKlgWdxGXp5jN5H7yMbX6IqKeeAqtrDgqRJ1dtQSsW7eOzz//nJiYGFJTUxk1ahSLFi1i5MiRDZVPOJjvtqdyPq+cR8a2o02MXAooLk8xmcj5+APK9vxO4Nhb5BJAIa7TVf9ySktLiYmJASA2NhZ3d3cpAMJqdhzOZuOedPq0D6N7m1C14wgbpRiNZH/4HuX79xE0/nYCRsh7khDX66olQFEUsrOzL64VoNPpLvl/uLDynxA3asfhbP69LplW0X7cO6K12nGEDcv9fBnl+/cRPHES/kNkgTEhbsRVS0BVVRUDBw685EM/MTHx4n9rNBqSk5Otl044hMKSaj7bdILWTfx4YkInnHQysltcmf+Qobg3j8X3pv5qRxHC7l21BBw7dqyhcggHZag18cGPx1AUhftGtMbFWVYFFH9mrqmh9Pdd+Pbrj2tkFK6RUWpHEqJRuGoJ0OnkDVlY16cbUjiVUcJDY9oS4u+hdhxhg8zVVWS++QZVp07iFtMMtyYyc6QQ9UWG1ArVHDxVwK5juQzrHk3PtmFqxxE2yFRZSebiRVSnnSFs6sNSAISoZ1IChCrS88r5cM0xooI9uaVfc7XjCBtkqqgg4/WF1KSfJ/zhaXh36aZ2JCEaHSkBosFVVht567vDaDUapo1rL+MAxGVVp6ViyMokYtp0vOIT1I4jRKNk8TBso9FIUlISGzZsAKC6uprq6mqrBRON1+ebT1BYWs2M8R1lSmDxJ4rZDIBn+440e2WBFAAhrMiiEnDq1ClGjBjBnDlzePbZZwHYtWsXzz33nFXDicZn97Ecdh/LZWTPprSM9lM7jrAxxuJizv/jRcqTDgDg5OurciIhGjeLSsBLL73EtGnT2Lx5M07/Pz1n9+7d2bdvn1XDicalqsbIV1tP0zTMm3H9mqkdR9iY2qIi0he8giEvD62HXCkiREOwqAScPHmSW265BeDiCl2enp5yOkDUybc/p1JWYeDOwXHoZKlX8V9qCwvIWPAKptJSombNxqOVzBopREOw6J04IiKC48ePX3LbkSNHiI6Otkoo0fj8fDCTbUmZDOwSRVyUnAYQ/2EsKyV93iuYKiqIevJp3FvEqR1JCIdh0dUBM2fO5OGHH2bSpEnU1tby0UcfsXz5cl588UVr5xONwInzepZvPkVspA93DGqhdhxhY3Re3vj06o1Xl64yD4AQDcyiEjBo0CCCg4NZsWIFnTt3Ji0tjddff51OnTpZO5+wc3nFVby3+hhe7k48dksHOQ0gLqrJykSj1eISFk7QLePVjiOEQ7KoBJSUlNCxY0c6duxo7TyiEak2GHnz28PU1pqZO7kzfl6uakcSNqImPZ2MRfNxCgikyV9evDjWSAjRsCz6WnbTTTfxyCOPsG7dOhkMKCxiMptZuj6F7MIKHhnbjqgQL7UjCRtRfe4s6QtfRePkRPjUR6QACKEii0rAli1b6N27N0uXLqVPnz48/fTTbN++HZPJZO18wk59uv4Ee5LzGNUrhvbNA9WOI2xE1ZkzZLw2H62rG1FPP4tLmKwZIYSaLCoBQUFB3HPPPXzzzTf88MMPNGvWjPnz59OvXz9r5xN2aNPedHYcyWZY92huvUnWBRD/UbR2NTpPT6KfeRaXkBC14wjh8Oq8dkBZWRllZWVUVFTg7u5ujUzCjqWc0/PVllMkxAUxvn+s2nGEjVAUBY1GQ/jUhzFVVePs7692JCEEFpaAtLQ01q5dy48//kh5eTnDhw9n0aJFdO7c2dr5hB3JzC9nycojBPm68eDotjjp5EoAARXHj6HfuJ6IadPRurmjdZMvD0LYCotKwG233caQIUP4y1/+Qu/evdHpZNU3cakag4l3fjiKVqth1u2dcHeVBSoFVBw5TNaSN3EODcNca0Dr5qZ2JCHEf7HonXrnzp24usrlXeLKlm1MIbuwkidv7yQrAwoAyg8mkf3eElwiIol68ml0XnKFiBC25oolYM2aNYwePRqA9evXX/EOxo0bV/+phF05mlbIrmO5jOrVVK4EEMCFApD17tu4RjchatZsdJ5SDIWwRVcsAStXrrxYAr755pvLbqPRaKQEOLhqg5EvNp0k0MeVMX1i1I4jbIRLWDheneIJvW8KOlkRUAibpVEURVE7RH0qLCzHbG5UT8mmBAd7k59fBlwY8f3uD0fZdyKfJyZ0omOsHAWoL/+9n+1J1ZlU3Jo1t4sJgOx1H9sT2cfWpdVqCAy8sdNsFg3fHj/+8vN633777Tf04MK+7T6ey74T+Yzt20wKgKDkl+2kv/JPSn75We0oQggLWTQw8MyZM5e9/ezZs/WZRdiRzIIKPt90gpgwb0b2lJXfHF3xti3kffEZHu074NOrj9pxhBAWumoJePbZZwGora29+N9/yMzMpHlzmQ3OEdUYTLyz8gg6rZZHx7XH2UnmA3Bk+s0byf/6SzzjEwh/+FG0zs5qRxJCWOiqJSA0NPSy/63RaGjfvj0jRoywXjJhsz5Zn0x2YSUzx3ckyE8mfnFkhvw88r/9Bq8uXS8sBuQk80MIYU+u+hf7xBNPABAfH8+AAQMaIo+wcet3prEnOY8xfWKIjwtSO45QmUtwCNFznsUtphkamURMCLtzxRKwf/9+unTpAoCnpyd79+697HbdunWzTjJhc05nlvDRqqO0ivbjZrkc0GEpikLhqpW4RkTi3b0H7rEt1I4khLhOVywBzz//PBs2bABg9uzZl91Go9Hw888/WyWYsC1llQaWrDyCt6cLj4xrj04r4wAckaIoFHz7DfqN6/Htn4h39x5qRxJC3IArloA/CgDA9u3bGySMsF2fbzpJeWUtC2b2w9dNzvs6IkVRyP96OcU/bcY3cSAhk+5SO5IQ4gZd19e5ffv2ceDAgfrOImzU0bRC9qbkMbp3DHHRsgSsI1IUhbwvPqP4p834DRlGyJ13o5GjQULYPYv+iu+++2727dsHwMcff8z06dOZOXMmH3zwgVXDCfXVGEx8+dMpgnzdGNmzidpxhIq0bm74Dx9J8O132MWMgEKIa7PouO7JkyeJj48H4Ouvv+azzz7D09OTyZMn89BDD1k1oFDXip9Pk11YyRMTOuHsJKO/HY1iMmHUF+EcFEzQ+AkAUgCEaEQsOhJgNpvRarWkp6djNBqJi4sjIiKC4uJia+cTKjqZXszWA5kkJkTKtMAOSDEayfnofc7/6x+YysvRaDRSAIRoZCw6EpCQkMDLL79MXl4eQ4YMASA9PR1/fzk/3FiZFYXvt6fi6ebE7YlyCZijUYxGst9/l/Kk/QRNmIjO68YWKRFC2CaLjgS8+uqruLi40KxZM2bMmAHA6dOnuesuGR3cWG1PyuRkRglj+jTD1UVOAzgSc62BrHfeojxpP8F3TCZgmMwMKkRjZdGRgICAAObMmXPJbYmJiSQmJlollFCXvqyGlb+mERPmzeCuUWrHEQ2saO0aKg4fIuSue/AbMFDtOEIIK7KoBBiNRt5//31Wr15Nbm4uoaGhjBkzhoceeghnWSyk0flqyymqaow8MLKNnAN2QAEjRuEW0wyv+AS1owghrMyiErBw4UIOHDjAc889R0REBFlZWbz77ruUlZUxd+5ca2cUDWhfSh57U/IY0aMJUSFyHthRmKurKPj+WwJvuQ2du7sUACEchEUlYP369axcuZKAgAAA4uLi6NChA2PHjpUS0IiYzGZW/HyaqGAvbrlJlol2FKbKSjIXL6I67QyenRLwbNde7UhCiAZi0cBAk8mE9n9mB9NoNCiKYpVQQh0b96STX1zNuH7NcNLJbHCOwFRRQcaiBVSfTSP8kcekAAjhYCx6px8+fDjTpk1j165dnD17lp07dzJ9+nSGDRtm7Xyigaj0ZsEAACAASURBVOQWVfLDr2l0aB5IgiwR7BBMZWVkLJyHISOdiEdn4N25i9qRhBANzKLTAXPmzOHtt9/m+eefJy8vj5CQEEaNGsX06dOtnU80kG+2nUan1XDfiNYyGNBBmGuqMRsMREx/HM/2HdSOI4RQgUZpZMf0CwvLMZsb1VOyutSsEv61bD9j+sQwrt/VxwIEB3uTn1/WQMkclzX3s6m8HK2HBxqtFsVkQqNzzHkg5LVsfbKPrUur1RAY+H/t3Xl8k1WiPvAnSdN9L91byg5lqVTKTgULylZAkRGpGyKLP0BGHBBkLsvcMiIggiAMg4PgguPIXEGoCF4WrSCLQLVAoUBbuqZbuq9Jk/P7g7FXZEtpkjfL8/18+Mw0hLxPX7Dvk/OenNO6Cdz3vB1w48YNPPvss+jXrx+mTp2KgoKCVh2MLE9dgxY7DlyBm7MDHu/LDYJsnbasDDmrElG6+18AYLcFgIhuumcJSExMRGBgIFatWgUfHx+89dZb5spFZvLFsQwUlNbi5fjucHU26O4QWSltaQny1qyCrqoK7jF9pY5DRBbgnj/1L168iO+//x7Ozs7o378/Ro/m8qG25HJ2OX74pQADewShdydOBrRlmuJi5L2zGvqGeoS9vhDO7fkRUCK6TwnQarVwdnYGALi7u6OxsdEsocj0auq12P51GgJ9XfHc412kjkMmJJqakL9+LfSaRoQtWATnthFSRyIiC3HPEqDRaPD+++83f93Q0HDL1wD4CQErJITAh19fRlWtBosSHoaLE28D2DKZgwMCnn0eDj6+cArlXhBE9H/u+dN/9OjRyM7Obv565MiRt3zNj5JZp29/ysXP10vx1NAO6BjqJXUcMpHG3Bw0FuTDs/9AuPWMkjoOEVmge5aAtWvXmisHmUleSQ3+5/tMRHX0w+gBHBa2VQ03biDv3bWQuzjDPboP5I6OUkciIgvEcWA7IoTARwevwNFBjhdHdYOcIzk2qT7jOvI3rIPc1RXhCxazABDRXXGBeDtyOq0IGflVeGpYR/h4OEkdh0yg/tpV5K9/Bwp3d4S/sQRKf3+pIxGRBeNIgJ2oa2jCnh8yEezniqEPhUgdh0yk/tpVKLy8EbZgEZQ+PlLHISILxxJgJ3YevILSygYsmNwbcjlvA9gavUYDuaMjfEaPhXfccMidXaSORERWwODbAadOncKyZcswe/ZsAMClS5dw5swZkwUj47meX4lzV4rxeN9wRLbzlToOGVlN6i/IWvIGGvPzIJPJWACIyGAGlYBdu3bhz3/+M4KCgnD69GkAgFKpxPr1600ajlqvvrEJf9t7EV7ujhg3qJ3UccjIalLOo2DzRjh4esHBy1vqOERkZQwqATt27MDOnTsxe/ZsyOU3/0jHjh2RmZlp0nDUent+yER5dSNemdATrs5KqeOQEVWfPYOCrZvhHBGBsAVvQOHeut3EiMj+GDQnoLa2FiEhNyeT/bpAkE6ng1LJi4olu5CpxuGzeRjaOwRdwvku0ZbUXU6D6u9/g3PHTgj94+tQuPAWABG1nEEjAX369MH27dtveWzXrl3o25c7kVmqmnotPtifhpA2bngmrrPUccjInDt1gu/osQh77U8sAET0wAwaCVi6dClmzZqF3bt3o7a2FmPHjoVSqcS2bdtMnY8eUNKPN1BTr8WCZ3rDyZF7xtuK6p/OwDWyOxTu7mgzcZLUcYjIyhlUAgIDA7Fnzx6cP38eKpUKQUFBiI6OhkLBi4slKiitxeGzeegXGYC2gR5SxyEjKT96GCWffQqfkaPg/4dnpI5DRDbA4HUCZDIZ+vTpY8osZCT//i4DDg4yJIzgFsG2ovzbQyj54p9w6x0NvyeekjoOEdkIg0pAXFzcXXcMPHLkiFEDUetczFLj5+ulmDCkPTzduGa8LSg7kITSL/8N95i+CJ4+CzIHrvFFRMZh0E+Tv/71r7d8XVxcjE8//RRjx441SSh6ME06PT7732to4+WMUf3bSh2HjEDfUI/K5O/h0X8AgqbNgIy34IjIiAwqAQMHDrzjYzNnzsTUqVMNOlBWVhYWL16MiooKeHt7Y/Xq1WjXrt0dn5uZmYknn3wSCQkJWLRokUGvT8DP10pRWFaHmeO7w0nJi4U1E0JA6PWQO7sg/M0/Q+HhCZmc+30RkXE98E8VZ2dn5ObmGvz85cuXIyEhAYcOHUJCQgKWLVt2x+fpdDosX74cI0aMeNBodqm+sQlfHLsOP09n9O0WIHUcagUhBG7s/BhFH+2A0Ovh4OXNAkBEJmHQSMD7779/y9cNDQ34/vvvMXjwYIMOolarkZaWhh07dgAA4uPjkZiYiLKyMvj63rqW/bZt2zBs2DDU1dWhrq7OoNcnYN+JLJRWNmDhM72h4AXDagkhUPLPXag4ehjeccOBu8zFISIyBoNKQHZ29i1fu7i4ICEhARMnTjToICqVCoGBgc0fKVQoFAgICIBKpbqlBFy5cgXHjx/Hxx9/jC1bthj6PdzCz8/+lk6tqG7EkXP5iIsJxyN9I0x+PH9/fuzQFIRej4ytH6Di6GGETBiHdi+9eNcJuWQc/LdsejzHlu2+JUCn02Hw4MEYPXo0nJycTBZEq9Vi6dKlWLVqVavWH1Cra6DXCyMms3x7kjOh0+sxPDoEJSXVJj2Wv7+HyY9hr4p2fYLKY0fgOyYe7V56EaWlNVJHsmn8t2x6PMemJZfLWv3G974lQKFQIDExEU888cQDHyQ4OBhFRUXQ6XRQKBTQ6XQoLi5GcHBw83NKSkqQk5ODmTNnAgCqqqoghEBNTQ0SExMf+Ni2rkmnx8lLhegS5o1gPzep41AreDzcBw5eXvAdO44jAERkFgbdDhg2bBi+//57DB069IEO4ufnh8jISCQlJWHChAlISkpCZGTkLbcCQkJCmrcpBoBNmzahrq6Onw64j/89m4vSygY8+xgXBrJGoqkJdelX4NajJ1wju8M1srvUkYjIjhhUAvR6PebOnYs+ffrc8u4dAFatWmXQgVasWIHFixdjy5Yt8PT0xOrVqwEAM2bMwLx589CrV68WRid1ZQP2n7iBqI5+eKhTG6njUAuJpiao/v431Px8HhErVsIpNFTqSERkZwwqAREREXj55ZdbdaCOHTti9+7dtz3+wQcf3PH5r776aquOZw8+/TYdTTo9nhnOXQKtjV6rgepvm1Gb+gv8pzzLAkBEkrhnCUhKSkJ8fDxee+01c+UhA+UV1+CXDDXGDWqHIF9XqeNQC+gbG1GwZRPqLl1EwPNT4T10mNSRiMhO3fMD5Xdb0Iek92VyJpwdFRgREyZ1FGqhml9SUJd2CYFTX2YBICJJ3XMkQAj7+qidtcgsqMLP10vxxJD28HDlJkHWxrPfADiFhsEplAWOiKR1zxKg1+tx6tSpe5aBO+0rQKZ1+GwunB0VeKxvuNRRyEC6ulqo/v43tHliIpzbd2ABICKLcM8SoNFo8Oc///muJUAmk3ErYTMrLq/D2fQSxD4UDBcnbilrDXQ1Nchb/w4a83LRVFEhdRwiomb3vIq4uLjwIm9BhBD4/Mh1AMDYAaZfHphar6m6CvnvroVGpULInFfhHtVb6khERM34VtKKXMkub54L4OvpLHUcug9ddTXy1q6GtqQYIa++BrcePaWORER0C04MtBJCCOw9ngVvd0eM7N9W6jhkALmLC5zCwxGQ8Bxcu0VKHYeI6Db3LAEpKSnmykH3cSGzDNfyKjFleGc4KR98gyUyPW2ZGjIHJRw8PRE84xWp4xAR3RU3nrcCQgjs/SETfp7OGBbNleUsmbakBLlrVkH1t/c5kkZEFo8lwAqcTS/BjcJqxA+KgNKBf2WWSlNUiNw1q6Cvq4f/5ATuBEhEFo8TAy1ck06PPcmZ8Pd2RmxUiNRx6C4aCwqQt24NoNMhfOEiOIVz3gYRWT6+rbRwh8/mobCsDlNGdIFczneWlkgIgeJPPwKEHmEsAERkRTgSYMFq6rXY/2MWerTzQW9uFWyxZDIZgma8AtFQD8dgjtYQkfXgSIAF++ZUNhoadXg6jlsFW6KGG1ko+mQnhF4PpY8PCwARWR2OBFio6joNjpzLQ0y3AIQHuEsdh36nPuM68jesg9zNDbrqKjh4eUsdiYioxVgCLJAQAv88fA06vcD4Ie2ljkO/U3/tKvI2vAsHT0+ELVzEAkBEVoslwAJdyFTjVFoR4ge1Q2gbN6nj0G/UXbmM/I3r4eDri/AFi+Dg7SN1JCKiB8YSYGGEENh9LAP+3s4YN4ibBFkcuRxOoWEImftHOHh5SZ2GiKhVODHQwly6UYb80lqMGRABpQOXB7YUWnUpAMC1S1eEL1nKAkBENoElwII06fT47H+voY2XMwb1DJY6Dv1H9flzyFqyCNVnTgMAVwIkIpvBEmBBzl8tQWFZHZ4a2pHLA1uI6p/OQLV1M5wj2sG1Zy+p4xARGRXnBFiQo+fy0MbLGX27BUgdhQBUnfwRhR9+AJdOnRH6x/mQO7tIHYmIyKj4dtNCXMurwNW8SgztHcLlgS2ARlVwswB07YbQ1/7EAkBENokjARagSafHx4fS4e3uiLiHw6SOQwAcg0MQPGs23HpFQe7kJHUcIiKT4EiABTh8Ng/5JbVIGNEFLk7sZVKqOHYU9deuAQA8YvqyABCRTWMJkFh9YxMOnslBlzAvxHAugKTKDn2D4l0foyL5mNRRiIjMgiVAYl9+n4nqWg0mDeskdRS7pv56P0p3/wvuMf0Q9OI0qeMQEZkFx54ldPlGGY6cz8Pwh8PQKYyLz0hBCAH1vr0o2/8VPPoPRNC06ZApuEgTEdkHlgCJCCHwxXcZ8PV0wqRhHaWOY7+EgEalguegIQicOg0yOQfHiMh+sARI5GpuBbILq/H8yK5wcuQ7T3MTQkBfXweFqxuCp88E5HIWACKyO/ypJ5Hvfy6Ai5MDBnQPlDqK3RF6PUr++Sly31oJXV0dZA4OLABEZJf4k08CtQ1anL9agr7dAviRQDMTej2KP/0IFUePwO2hhyB34SJARGS/eAWSwDencqBp0mNYdIjUUeyK0OtRtHM7qn48Ad+x4+D3xERuBkREdo0lwMzySmpw6EwO+nYLQLsgT6nj2BX1V3tQ9eMJ+E14En7jJkgdh4hIciwBZrb7WAYcHORIeKyL1FHsjnfcCDj4+sF76DCpoxARWQTOCTCjvOIaXMxUY2TfcHi5OUodxy7otVqUHTwA0dQEBy8vFgAiot/gSICZCCHw2eGrcHNRYkRMuNRx7IJeq4Fqy/uovZAKp7AwuPWMkjoSEZFF4UiAmVzILMOVnAqMGRABdxel1HFsnr6xEQUb30PtxQsIeGEqCwAR0R1wJMBMvjmVDS93Rwzvw62CTU3f0ID8TRtQfzUdgVNfhtfgIVJHIiKySBwJMIOcomqk51ZgRJ8wKB14yk1NW1KMxtxcBE2fxQJARHQPHAkwMSEEvkzOhJNSgaG9Q6WOY9P0Wi3kSiWcwtui/dtroHB1kzoSEZFF49tSE7tRWI3UDDXiB3EugCnpamqQu2olyv/3EACwABARGYAjASZ2LCUfcpkMsVFcHdBUmqqrkLduLbSFKigDg6SOQ0RkNVgCTKiwrA4nUlUY9nAoPLkugEk0VVYgb90aaEtLEfLqa3Dr0VPqSEREVoMlwIROXSqEADB2QITUUWySXqtB3trV0JaXIXTefLh2i5Q6EhGRVWEJMBGNVodjKfno3s4Hvp7OUsexSXKlI7wfGwmnkBC4dOYyzERELcUSYCLn0ktQXafF6P4cBTA2bUkJtGVquHbtxmWAiYhagZ8OMAG9XuDbn3Lh5+mMyHY+UsexKZqiQuSuWYXC7dug12qljkNEZNVYAkzgVFohsouq8eQj7SHnfvVG01hQgNw1b0NotQh9dT7kSn7kkoioNXg7wMiEEDh4OgchbdwwsAc/rmYsjfl5yHtnDSADwhYuhlMoF14iImotjgQYWVp2OfJKajGqX1vIOApgNJXJ3wMKOcLfeJMFgIjISDgSYGSHzuTA3UWJ/t0DpI5iE4QQkMlk8J88BT6jxkDpwzkWRETGwpEAIyoqr8PFzLL/bBSkkDqO1au/fg05iSugLS+HTC5nASAiMjKWACM6ebEQMgCDenEuQGvVXU1H3vp10Dc2AEIvdRwiIpvE2wFGohcCyb8UILKdD9p4uUgdx6rVXU5D/qYNUPr6IWzBIjh4e0sdiYjIJnEkwEguZ5ejokaDQT05CtAadVfTkb9xPZT+AQhbuJgFgIjIhDgSYCRHzubB01WJmK6cENgaTiGhcO8Tg4DJCVB4eEgdh4jIpnEkwAgqahrxy/VSDOwZBEclJwQ+iLqr6dBrtVC4uyN4+iwWACIiM2AJMILkXwogAAzqGSx1FKtUfeY08t5ZjbKkfVJHISKyK7wd0Ep6vcDRc3mI6uiH8AB3qeNYnaqTJ1D44T/g0rkLfEePkToOEZFdYQlopfTcClTVadEvknMBWqryeDKKPtoBl67dEPrqa5A7OUkdiYjIrrAEtNLeHzLh7qJEny4sAS2hq61Fye5/wbV7D4TMmQe5o6PUkYiI7A5LQCtk5FfiWl4lnn60E5wcOSGwJRRubghftARKf3/IlSwARERSYAlohaPn8+DipMDQ3iFSR7EaZQcPQDQ1wS9+PJxCuBEQEZGU+OmAB9So0eFsegn6RwbCxYldyhDqpH0o/fcX0OTnQei5FDARkdR49XpAZ9OLoW3SY0APrhB4P0IIqL/ag7KkffAYOAhBL02HTM7+SUQkNZaAByCEwA+/FMDX0wmdw7ykjmPxSr/8N8q/+RqeQx5B4AtTWQCIiCwEfxo/gNziGlzNq8SIPuGQyWRSx7F4joFB8Hp0OAsAEZGF4UjAAzh8Ng+OSjk3C7oHoddDoyqAU2gYvIbEwmtIrNSRiIjod/i2rIXqG5tw+nIR+kcGwtONH227E6HXo+iTnchJXAFNUaHUcYiI6C5YAlro1wmBQ6K4T8CdCL0eRTu2o+qHZPiMGgNlQKDUkYiI6C54O6CFjqeqEOjjgk6hnBD4e0KnQ+H2bag+cxp+E56E37gJUkciIqJ74EhAC6jUtbiWV4nBvYI5IfAOqk4cR/WZ02jz1NMsAEREVoAjAS1wLCUfcpmMtwLuwnNILBz8/ODWo6fUUYiIyAAcCTCQtkmPExcK0TcyAN7u3O3uV3qNBoU7P4S2tAQyuZwFgIjIirAEGOhilhr1jU3cMvg39I2NKNj0HqpO/ID6jOtSxyEiohbi7QADHTufD083R/Tq4Cd1FIugb2hA/sb1qL92FUEvTYdn/4FSRyIiohYyWwnIysrC4sWLUVFRAW9vb6xevRrt2rW75TmbN2/GgQMHIJfLoVQqMX/+fMTGSr/IjEpdi4tZZXgitj0cFBw80dXXI3/DOjRkZSJo+ix49h8gdSQiInoAZisBy5cvR0JCAiZMmICvvvoKy5Ytw8cff3zLc6KiojBt2jS4uLjgypUreO6553D8+HE4OzubK+Yd/XixEDIAsVHcMhgAIG7uABg86//Bo09ficMQEdGDMsvbWrVajbS0NMTHxwMA4uPjkZaWhrKyslueFxsbCxcXFwBA165dIYRARUWFOSLelU6vx6lLReje3hc+HvY9IVBXUwNdYyMUrm4IX7SEBYCIyMqZZSRApVIhMDAQCoUCAKBQKBAQEACVSgVfX987/pm9e/eibdu2CApq2fr8fn7urc77Wyd+KYC6qgHTn+gJf38Po762NdFUVOLS+jUoDwpE5JLFUsexC/b8781ceI5Nj+fYslnkxMAzZ87gvffew4cfftjiP6tW10CvF0bLcvDHLPh4OKFTkDtKSqqN9rrWpKmiAnnr1kCrLkX7l1+y2/NgTv7+HjzPJsZzbHo8x6Yll8ta/cbXLLcDgoODUVRUBJ1OBwDQ6XQoLi5GcPDti+6kpKRg4cKF2Lx5Mzp06GCOeHdVXafBxSw1+nYLgMJOt8DVlpUhd+0qaMvUCP3j6/B+KErqSEREZCRmubL5+fkhMjISSUlJAICkpCRERkbedisgNTUV8+fPx8aNG9GjRw9zRLunM5eL0aQTGNzLPlcIFEJAtXUzdJWVCJu/AK5du0kdiYiIjEgmhDDe2Pk9ZGRkYPHixaiqqoKnpydWr16NDh06YMaMGZg3bx569eqFp556Cvn5+QgM/L+d59asWYOuXbsafBxj3g5Yves8quu1SHy5n93uFdCYmwO9tgku/xmV4fCeefA8mx7PsenxHJuWMW4HmK0EmIuxSkB+SQ2Wbj+DiY90QPygdq0PZkU0hYWoSTkPn1Gjbys//I/aPHieTY/n2PR4jk3LGCXAIicGWoIzl4shA+zuVkBjQT7y1q0B9Hp4DhoMBy9umUxEZKtYAu5ACIGz6cXoFOZlV2sDNObmIu/dNYBcjrCFi1kAiIhsnH1Oeb+PzIIqqNR1djUK0JB9A7nvvA2ZgwPCF74Jp5BQqSMREZGJcSTgDn66UgwHhQwxXe1nx0BtcTEUrq4IfX0hHP3t5/smIrJnLAG/I4TAhUw1Ood5w9XZ9k+Prr4eChcXePTtB7fevSFXOkodiYiIzIS3A34np6gGKnUdYrrZ/rvhuvQryFq8ALVplwCABYCIyM6wBPzO6ctFkMmAPl38pY5iUnWX05D/3rtw8PLi/X8iIjtl++PdLaBt0uPM5SJ0j/CBp5vtviuuvZiKgs2boAwIRNif3oCDp6fUkYiISAIcCfiNc1eLUVbViOEx4VJHMZnGvFwUvL8RjsEhCF+4mAWAiMiOcSTgNw6fzUOAtwuiOvpJHcVkHEPD4PfkU/Aa8ggUbm5SxyEiIglxJOA/isrrkFlQhdiHgiG3wX0Cqs/9BE1REWQyGXxHjmYBICIiloBfnU8vAQCb/FRA5YnjUG3dAvW+PVJHISIiC8LbAf9x7moJIoI8EOjjKnUUo6pM/h5Fn+yEa7fuCHzhJanjEBGRBeFIAIAGTROyC6vRo52v1FGMquLYERR9vAOuPXoi5NU/Qu5kP/sgEBHR/XEkAMAv19XQ6QV6tredEiB0OlSdPgW33tEInjUbcqVS6khERGRhWAIAnLlcBG93R3QJ95Y6ilGIpibIHBwQ9trrkDkoIXPgXzMREd3O7m8H1Dc24UKmGjHdAiCXW/+nAtT7v0Leu2uh12ggd3ZhASAioruy+xKQcq0ETTqBft0CpY7SKkIIlO79H6i/2gOlXxte/ImI6L7s/kqRmqGGp6sSHUOtd+U8IQRK//0Fyg99A8/YRxD4/FTI5Hbf74iI6D7s+kqh0erw8/VS9O7cBjIrXiCobP9XKD/0DbwejWMBICIig9n1SEB6bgU0Wj0e7mLdCwS5x/SDEAJ+45+w6jJDRETmZddvGVMz1HBQyNGtrfV9KkDo9aj+6QyEEHAKCUGbCU+yABARUYvYbQnQ6wV+ulyEhzr6wVGpkDpOiwidDoU7/gHV37egLu2S1HGIiMhK2e3tgOyialTVafFwV3+po7SIaGpC4fZtqP7pDPyemAi3Hj2ljkRERFbKbktA2o0yAEBkhI/ESQwnmpqg+vvfUJNyDm0mPQ3fUWOkjkRERFbMbkvAhcwyhPm7wdvdetbTb8jKRE3qz/B/JgE+Ix6XOg4REVk5uywBeiFwo7AKsVEhUkcxiBACMpkMLp27oN3KVXD0t+5PMxARkWWwy4mBWaoqaLR6tAvykDrKfekbG5G/YR2qz50FABYAIiIyGrssAafTiuCgkCG6s2VPCtQ31CN/wzrUpV2C0GikjkNERDbGLm8HXMoqQ2SEL1ydLffb19XVIf+9d9GQlYngGa/Ao19/qSMREZGNsbuRAHVlA1TqOnQJ95I6yl3pGxuR9+5aNNzIQvArc1gAiIjIJCz3rbCJXM2rAAD06uAncZK7kzk6wrVbJPzGTYD7Q72ljkNERDbK7kpAek45XJwcEOrvJnWU2zRVVkJfVwvH4BD4T3pa6jhERGTj7Op2gF4IXMgsQ7e23lBY2E57TRXlyFv7NvI3boBoapI6DhER2QHLuhKaWGZ+FcqrG9HHwpYK1papkbvmbWjLyxH40suQOdjdAA0REUnArq42564WQyGXoXcnyykB2tIS5L2zBrraGoS9vgAuHTtJHYmIiOyEXZWAn6+Voltbb4v6aGDpV3ugq6tF2OsL4dy+g9RxiIjIjljO1dDESivrUVRej7iHw6SOcovAZ1+AdtRYOIWGSh2FiIjsjN3MCbihqgYAdAj1lDgJ0Jifj4LNm6BvqIfc2ZkFgIiIJGE3IwEXMtVwdlQgIlDa/QIac3OQt24toFCgqbIKjs4ukuYhIiL7ZRclQAiBq7kV6BLuDQeFdIMfDTduIO/dtZA7OSFswRtwDAyULAsREZFd3A4oqbg5H0DKVQLrMzORt2415C7OCH/jTTgGBkmWhYiICLCTkYDL2eUAgK5tvSXLoHBzg1N4WwS9PBNKP8tdspiIiOyHXYwEpOdUwMvdEaFtzL9UsEZVACEEHAMDEbZwMQsAERFZDLsoAdfyKtE51Asymcysx61Nu4TsxBUoP3gAAMx+fCIionux+RJQXt0IdVUDOoWZ91ZATeovKNi4HsqAQHgOiTXrsYmIiAxh83MCsgtvrg8QEehutmPWpJxHwdbNcAoNQ9jrC6FwN9+xiYiIDGXzJSAtuwxKBzk6hJhnkaCm6iqo/vF3OEdEIPS1P0HhanlbFhMREQF2UAKu51WiY4gnlA4KsxzPwcMToa++BqeIdlC4cCEgIiKyXDY9J6CmXovsomp0CTf9fIDKE8dRdfokAMC1WyQLABERWTybLgE5RdUQAuhs4kmBFcnfoWjndlSdPAkhhEmPRUREZCw2fTsgo6AKMgBtTTgpsOLoYRR/9incekUhePZcfgyQiIishk2XgOt5lQjyc4WHq6NJXr/824Mo+eJzuPWORvCs2ZArlSY5DhERkSnY7O0AIMQUQgAAEO1JREFUvV7gen6FSW8F6Gpr4R7TFyGvzGEBICIiq2OzIwEqdS3qG3XoFOpl1NcVQkBXVQkHL2/4PTEREAIyuc12KSIismE2e/VKu3Fz06BuRtw0SAgB9Z7/QfbypdCq1ZDJZCwARERktWz2CpZdVA1vd0e08TbOR/WEECjd/S+UHUiC+8N94ODjY5TXJSIikorN3g64nleJtoEeRnktIQRK/rkLFUcPwztuOPynPMdPARARkdWzyZGAsqoGFFfUo0c7X6O8XuWxI6g4ehg+j41kASAiIpthkyMBOcU1AIy3PoDnkEcgUypv/i8LABER2QibHAnI+08JCA948NsBQqeDet9e6OpqIXd0hFfsUBYAIiKyKTZZArKLquHv7QxX5wcb6BBNTVB9sBXqfXtRk5Ji5HRERESWwSZvB+QW1SDM/8FuBYimJhT8fQtqU86jzR8mw2vwECOnIyIisgw2NxJQ39iE4op6dAjxbPGf1Ws1KNiyCbUp5+E/5Vn4jhxtgoRERESWweZGAgrL6gAAoQ8wEqCrqUVjQT4Cnn8R3kMfNXY0IiIii2JzJaCgtBYA0DbA8BKgb2yETKmE0scH7f7yV8idnEwVj4iIyGLY3O0AlboWbs4O8PEw7EKuq69H/oZ1KP7sUwBgASAiIrthgyWgDuEB7gZ9nE9XV4v89e+gPuM6XLt2M0M6IiIiy2FzJaBAXYuIoPuvD6CrqUHeurVoyL6BkP83Bx59+5khHRERkeWwuTkBOp1AiJ/bPZ8jhED+xvXQ5OchZM6rcI/qbaZ0RERElsPmSgAA+N9n50CZTAa/8RMAmRxuPXqaKRUREZFlsckSEOTnesfHmyrKUX/9Gjxi+sGtZ5SZUxEREVkWmysBSqUcnm6Otz2uLVMj75010FVXwbVbdyjcjbO5EBERkbWyuRIQ6O0K+e8+GaAtKUHuutXQ19Yi9LU/sQAQERHBBkvA728FaIqKkLduNfQNjQj70yI4t2snTTAiIiILY3MlIMDn1kmBtRd+gdBoEb5wEZzC20qUioiIyPLYXAlo4+kMABA6HWQKBXxGPA6PfgPg4NnyDYWIiIhsmc0tFuTt4YTG3BzcWLYEDTnZAMACQEREdAdmKwFZWVmYPHkyRo4cicmTJ+PGjRu3PUen0+Evf/kLRowYgcceewy7d+9u8XHcq0qQu3Y1hFYLuZOzEZITERHZJrOVgOXLlyMhIQGHDh1CQkICli1bdttz9u/fj5ycHHz77bf417/+hU2bNiEvL69Fx6n8ZDvkri4If+NNOAYGGis+ERGRzTHLnAC1Wo20tDTs2LEDABAfH4/ExESUlZXB19e3+XkHDhzAH/7wB8jlcvj6+mLEiBE4ePAgpk+fbvCxnENDEfqHKXDw9jb690E3yeX335yJWo/n2fR4jk2P59h0jHFuzVICVCoVAgMDoVAoAAAKhQIBAQFQqVS3lACVSoWQkJDmr4ODg1FYWNiiY3VfvNA4oemu/Py4zoI58DybHs+x6fEcWzabmxhIREREhjFLCQgODkZRURF0Oh2AmxMAi4uLERwcfNvzCgoKmr9WqVQICgoyR0QiIiK7Y5YS4Ofnh8jISCQlJQEAkpKSEBkZecutAAAYNWoUdu/eDb1ej7KyMhw+fBgjR440R0QiIiK7IxNCCHMcKCMjA4sXL0ZVVRU8PT2xevVqdOjQATNmzMC8efPQq1cv6HQ6/Pd//zdOnDgBAJgxYwYmT55sjnhERER2x2wlgIiIiCwLJwYSERHZKZYAIiIiO8USQEREZKdYAoiIiOwUSwAREZGdsroSYK7dCO2ZIed48+bNGDt2LMaNG4eJEyfihx9+MH9QK2fIef5VZmYmHnroIaxevdp8AW2Aoef4wIEDGDduHOLj4zFu3DiUlpaaN6gVM+Qcq9VqzJw5E+PGjcPo0aOxYsUKNDU1mT+slVq9ejXi4uLQtWtXXL169Y7PeeDrnrAyzz//vNi7d68QQoi9e/eK559//rbn7NmzR0ybNk3odDqhVqtFbGysyM3NNXdUq2XIOU5OThZ1dXVCCCEuX74s+vTpI+rr682a09oZcp6FEKKpqUk899xz4vXXXxdvv/22OSNaPUPOcWpqqhg9erQoLi4WQghRVVUlGhoazJrTmhlyjleuXNn8b1ej0YhJkyaJr7/+2qw5rdlPP/0kCgoKxKOPPirS09Pv+JwHve5Z1UjAr7sRxsfHA7i5G2FaWhrKyspued7ddiOk+zP0HMfGxsLFxQUA0LVrVwghUFFRYfa81srQ8wwA27Ztw7Bhw9CuXTszp7Ruhp7jnTt3Ytq0afD39wcAeHh4wMnJyex5rZGh51gmk6G2thZ6vR4ajQZarRaB3OrdYDExMbcts/97D3rds6oScK/dCH//vNbuRmivDD3Hv7V37160bduW+zy0gKHn+cqVKzh+/DimTp0qQUrrZug5zsjIQG5uLp599lk8+eST2LJlCwTXUDOIoed49uzZyMrKwpAhQ5p/9enTR4rINutBr3tWVQLI8pw5cwbvvfce1q1bJ3UUm6PVarF06VL85S9/af4hS8an0+mQnp6OHTt24JNPPkFycjK++uorqWPZlIMHD6Jr1644fvw4kpOTcfbsWY7OWgirKgHcjdD0DD3HAJCSkoKFCxdi8+bN6NChg7mjWjVDznNJSQlycnIwc+ZMxMXF4aOPPsIXX3yBpUuXShXbqhj6bzkkJASjRo2Co6Mj3N3dMXz4cKSmpkoR2eoYeo4//fRTjB8/HnK5HB4eHoiLi8Pp06eliGyzHvS6Z1UlgLsRmp6h5zg1NRXz58/Hxo0b0aNHDymiWjVDznNISAhOnz6No0eP4ujRo3jxxRfx9NNPIzExUarYVsXQf8vx8fE4fvw4hBDQarU4deoUunXrJkVkq2PoOQ4LC0NycjIAQKPR4OTJk+jcubPZ89qyB77uGXUKoxlcv35dTJo0STz++ONi0qRJIiMjQwghxPTp00VqaqoQ4uZs6mXLlonhw4eL4cOHi88//1zKyFbHkHM8ceJE0b9/fzF+/PjmX1euXJEyttUx5Dz/1saNG/npgBYy5BzrdDrx1ltviVGjRokxY8aIt956S+h0OiljWxVDznF2draYOnWqiI+PF6NHjxYrVqwQWq1WythWJTExUcTGxorIyEgxaNAgMWbMGCGEca573EWQiIjITlnV7QAiIiIyHpYAIiIiO8USQEREZKdYAoiIiOwUSwAREZGdYgkgsmALFizApk2bpI5xXyNHjsTZs2fv+vvTpk3Dvn37zJiIiAzhIHUAInsQFxeH0tLSW5b/PXjwoCSbqCxYsAAHDx6EUqmEUqlEz549sXTpUrRv3/6BX/PQoUPN/3/9+vUoKirC22+/3fzYhx9+2KrMd9LU1IQePXrAxcUFMpkMHh4eGDt2LBYuXAi5/P7vb3788Uf813/9F44ePWr0bETWgiMBRGaydetWpKSkNP+Sche1WbNmISUlBd999x28vLywZMkSybK0VlJSElJSUvDRRx9h37592LNnj9SRiKwGSwCRhPR6PebNm4fBgwcjJiYGzz//PDIyMu74XLVajRkzZiAmJgb9+vXDs88+2/x7hYWFmDNnDgYMGIC4uDjs2rXLoOO7urpi7NixuHbtGgCgsbERK1euxJAhQxAbG4tVq1ZBo9Hc9/iPPPIITp8+jWPHjmH79u3Yv38/oqOjMXHiRADAlClT8OWXX6KhoQEPP/zwLd9jSUkJoqKiUF5eDgA4cuQIxo8fj5iYGEyZMgVXr1416Htp3749oqOjcfny5ebHdu/ejdGjRyM6OhojRozA7t27AQDV1dV45ZVXUFBQgOjoaERHR0OtVkOv12Pr1q0YMWIE+vfvj/nz56OystKg4xNZI5YAIokNGzYMhw4dwokTJ9C5c2csXLjwjs/7xz/+gfDwcJw8eRLHjx/Ha6+9BuBmkZg1axZ69eqF5ORk7NixA9u3b8fJkyfve+yamprm9d4BYPPmzbh48SL27duHvXv34vz589i2bds9j/9bjz76KF5++WWMGzcOKSkp+PLLL2/5fWdnZ4wYMQJff/1182MHDhzAwIED4ePjg9TUVCxduhQrV67E6dOn8dRTT2H27NnNReReMjIycP78eURERDQ/5ufnh23btuH8+fNITExEYmIirly5Ag8PD2zduhUhISHNIzN+fn7YuXMnvvvuO+zatQvJyclwdXXFypUr73tsImvFEkBkJnPmzEFMTAxiYmIwe/ZsAIBcLsfEiRPh7u4OJycnzJ07F5cuXUJdXd1tf16pVKK4uBgqlQqOjo7o27cvgJu7OdbU1OCVV16Bo6MjIiIi8NRTT91yof29bdu2ISYmBqNGjYJGo8Fbb70FANi/fz/mzp0LX19f+Pn5Yc6cOc3b6t7t+C0VHx9/S7akpCTEx8cDAL744gskJCQgKioKCoUCkyZNAgBcuHDhrq83fvx49O7dG2PGjMGgQYMwefLk5t+Li4tDeHg4ZDIZBg4ciIEDB+LcuXN3fa3PP/8cr7/+OgIDA5v/Pg4ePAi9Xv9A3yuRpePEQCIz2bx5MwYNGnTLYzqdDuvWrcOhQ4dQXl7ePKGtvLwcrq6utzx35syZ2LhxI6ZOnQq5XI5nnnkG06dPR0FBAVQqFWJiYm553f79+981y8yZM/Hqq6/e9nhxcTFCQkKavw4JCUFRUdE9j99SgwYNQlVVFS5evAhPT09cv34dw4cPBwAUFBRg//792LlzZ/PztVptc4Y72bdvH0JCQnDgwAFs2LAB9fX1cHR0BAAcO3YMW7ZsQXZ2NvR6PRoaGtCrV6+7vlZBQQFeeeWV2yYWqtVq+Pv7t/h7JbJ0LAFEEtq7dy+Sk5Px0UcfITQ0FOXl5Rg4cCDutK+Xu7s7lixZgiVLliA9PR0vvPACoqKiEBwcjIiICHzzzTetzhMQEICCggJ06NABwM09yX+dwHi34/fr1++W15DJZPc8hoODA0aNGoWvv/4a7u7uiIuLay48QUFBmDNnDmbOnNmi3HK5HPHx8Th8+DC2bt2KRYsWoaGhAfPmzcP69esxdOhQKJVKzJo1q/nc3ilnUFAQ1q1bh4ceeqhFxyeyVrwdQCSh2tpaODo6wtvbG/X19diwYcNdn3v06FHk5ORACAEPDw8oFArI5XL07t0bSqUSH374IRobG6HT6ZCeno6LFy+2OM/YsWOxefNmlJWVoaysDFu2bMH48ePvefzfa9OmDfLz8+9YZH4VHx+PAwcOICkpCePGjWt+/Omnn8Znn32G1NRUCCFQW1uLo0eP3vH2yJ3MnDkTn3/+OcrKyqDRaKDVauHj4wOFQoFjx47dMk/Cz88P5eXlqKmpaX7smWeewbvvvouCggIAN0cAjhw5YtCxiawRSwCRhCZOnIiAgADExsYiPj4e0dHRd31uVlYWXnzxRURHR2PKlCl44YUXEBMTAwcHB3zwwQdITU1FXFwcBgwYgOXLl99ycTPU3Llz0a1bN4wbNw7jx49HVFQUZs2adc/j/96YMWOg1WrRr1+/5nv6v/fwww9DoVCgrKwMgwcPbn68d+/eWLFiBVasWIG+ffti5MiRLVpkqHv37ujduze2b98OT09PvPnmm5g7dy769euHQ4cOYdiwYc3P7dKlCx5//HEMHz4cMTExUKvVeOmllxAbG4upU6ciOjoazzzzzD3nIxBZO5m4V10nIiIim8WRACIiIjvFEkBERGSnWAKIiIjsFEsAERGRnWIJICIislMsAURERHaKJYCIiMhOsQQQERHZqf8PDdGFXE3XkzUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\"\"\"\r\n",
    "base_params_lgb = {\r\n",
    "                    'boosting_type': 'gbdt',\r\n",
    "                    'objective': 'binary',\r\n",
    "                    'metric': 'auc',\r\n",
    "                    'learning_rate': 0.01,\r\n",
    "                    'num_leaves': 14,\r\n",
    "                    'max_depth': 19,\r\n",
    "                    'min_data_in_leaf': 37,\r\n",
    "                    'min_child_weight':1.6,\r\n",
    "                    'bagging_fraction': 0.98,\r\n",
    "                    'feature_fraction': 0.69,\r\n",
    "                    'bagging_freq': 96,\r\n",
    "                    'reg_lambda': 9,\r\n",
    "                    'reg_alpha': 7,\r\n",
    "                    'min_split_gain': 0.4,\r\n",
    "                    'nthread': 8,\r\n",
    "                    'seed': 2020,\r\n",
    "                    'silent': True,\r\n",
    "}\r\n",
    "\r\n",
    "\"\"\"使用训练集数据进行模型训练\"\"\"\r\n",
    "final_model_lgb = lgb.train(base_params_lgb, train_set=train_matrix, valid_sets=valid_matrix, num_boost_round=13000, verbose_eval=1000, early_stopping_rounds=200)\r\n",
    "\r\n",
    "\"\"\"预测并计算roc的相关指标\"\"\"\r\n",
    "val_pre_lgb = final_model_lgb.predict(X_val)\r\n",
    "fpr, tpr, threshold = metrics.roc_curve(y_val, val_pre_lgb)\r\n",
    "roc_auc = metrics.auc(fpr, tpr)\r\n",
    "print('调参后lightgbm单模型在验证集上的AUC：{}'.format(roc_auc))\r\n",
    "\"\"\"画出roc曲线图\"\"\"\r\n",
    "plt.figure(figsize=(8, 8))\r\n",
    "plt.title('Validation ROC')\r\n",
    "plt.plot(fpr, tpr, 'b', label = 'Val AUC = %0.4f' % roc_auc)\r\n",
    "plt.ylim(0,1)\r\n",
    "plt.xlim(0,1)\r\n",
    "plt.legend(loc='best')\r\n",
    "plt.title('ROC')\r\n",
    "plt.ylabel('True Positive Rate')\r\n",
    "plt.xlabel('False Positive Rate')\r\n",
    "# 画出对角线\r\n",
    "plt.plot([0,1],[0,1],'r--')\r\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"保存模型到本地\"\"\"\r\n",
    "# 保存模型\r\n",
    "import pickle\r\n",
    "pickle.dump(final_model_lgb, open('model_lgb_best.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
